2013-04-03 10:33:48.318121 7fce0612e780 2 mon.hotcpc9039@-1(probing) e5 init 2013-04-03 10:33:48.318189 7fce0612e780 10 mon.hotcpc9039@-1(probing) e5 bootstrap 2013-04-03 10:33:48.318195 7fce0612e780 10 mon.hotcpc9039@-1(probing) e5 unregister_cluster_logger - not registered 2013-04-03 10:33:48.318197 7fce0612e780 10 mon.hotcpc9039@-1(probing) e5 cancel_probe_timeout (none scheduled) 2013-04-03 10:33:48.318199 7fce0612e780 0 mon.hotcpc9039@-1(probing) e5 my rank is now 0 (was -1) 2013-04-03 10:33:48.318207 7fce0612e780 10 mon.hotcpc9039@0(probing) e5 reset_sync 2013-04-03 10:33:48.318210 7fce0612e780 10 mon.hotcpc9039@0(probing) e5 reset 2013-04-03 10:33:48.318212 7fce0612e780 10 mon.hotcpc9039@0(probing) e5 timecheck_finish 2013-04-03 10:33:48.318227 7fce0612e780 10 mon.hotcpc9039@0(probing) e5 cancel_probe_timeout (none scheduled) 2013-04-03 10:33:48.318230 7fce0612e780 10 mon.hotcpc9039@0(probing) e5 reset_probe_timeout 0x1b91350 after 2 seconds 2013-04-03 10:33:48.318258 7fce0612e780 10 mon.hotcpc9039@0(probing) e5 probing other monitors 2013-04-03 10:33:48.318864 7fce027bf700 10 mon.hotcpc9039@0(probing) e5 ms_get_authorizer for mon 2013-04-03 10:33:48.319453 7fce027bf700 10 mon.hotcpc9039@0(probing) e5 ms_get_authorizer for mon 2013-04-03 10:33:48.320796 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 do not have session, making new one 2013-04-03 10:33:48.320804 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 ms_dispatch new session MonSession: mon.2 10.252.24.88:6789/0 is open for mon.2 10.252.24.88:6789/0 2013-04-03 10:33:48.320811 7fcdffcdb700 5 mon.hotcpc9039@0(probing) e5 setting monitor caps on this connection 2013-04-03 10:33:48.320814 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 handle_probe mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc12533 quorum 0,2 paxos( fc 427161 lc 427188 )) v4 2013-04-03 10:33:48.320819 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 handle_probe_reply mon.2 10.252.24.88:6789/0mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc12533 quorum 0,2 paxos( fc 427161 lc 427188 )) v4 2013-04-03 10:33:48.320827 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 monmap is e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} 2013-04-03 10:33:48.320840 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 peer name is hotcpc12533 2013-04-03 10:33:48.320844 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 existing quorum 0,2 2013-04-03 10:33:48.320846 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 peer paxos version 427188 vs my version 427188 (ok) 2013-04-03 10:33:48.320849 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 start_election 2013-04-03 10:33:48.320850 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 cancel_probe_timeout 0x1b91350 2013-04-03 10:33:48.320872 7fcdffcdb700 0 log [INF] : mon.hotcpc9039 calling new monitor election 2013-04-03 10:33:48.320906 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(0) start -- can i be leader? 2013-04-03 10:33:48.320980 7fcdffcdb700 1 mon.hotcpc9039@0(electing).elector(350) init, last seen epoch 350 2013-04-03 10:33:48.320985 7fcdffcdb700 10 mon.hotcpc9039@0(electing).elector(350) bump_epoch 350 to 351 2013-04-03 10:33:48.321390 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 do not have session, making new one 2013-04-03 10:33:48.321394 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 ms_dispatch new session MonSession: mon.0 10.252.24.25:6789/0 is open for mon.0 10.252.24.25:6789/0 2013-04-03 10:33:48.321398 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 setting monitor caps on this connection 2013-04-03 10:33:48.373542 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(351) handle_ack from mon.2 2013-04-03 10:33:48.373548 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(351) so far i have {0=4294967295,2=4294967295} 2013-04-03 10:33:48.720343 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:48.720356 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 mesg 0x20add80 from 10.252.24.88:6789/0 2013-04-03 10:33:48.780953 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:48.780964 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:33:49.821606 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:49.821617 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 mesg 0x20fc480 from 10.252.24.88:6789/0 2013-04-03 10:33:52.477502 7fcdfecd9700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.58.31:6800/10908 mds protocol 0 2013-04-03 10:33:52.477863 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 36 bytes epoch 5) v1 until we get in quorum 2013-04-03 10:33:53.318324 7fce004dc700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 36 bytes epoch 5) v1 until we get in quorum 2013-04-03 10:33:53.321435 7fce004dc700 5 mon.hotcpc9039@0(electing).elector(351) election timer expired 2013-04-03 10:33:53.321444 7fce004dc700 10 mon.hotcpc9039@0(electing).elector(351) bump_epoch 351 to 352 2013-04-03 10:33:53.331846 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 win_election, epoch 352 quorum is 0,2 features are 4294967295 2013-04-03 10:33:53.331861 7fce004dc700 0 log [INF] : mon.hotcpc9039@0 won leader election with quorum 0,2 2013-04-03 10:33:53.332187 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish 2013-04-03 10:33:53.332193 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:33:53.332196 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: mds.? 10.252.58.31:6800/10908 is open for mds.? 10.252.58.31:6800/10908 2013-04-03 10:33:53.332201 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:33:53.332245 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 resend_routed_requests 2013-04-03 10:33:53.332249 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 register_cluster_logger 2013-04-03 10:33:53.332255 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start 2013-04-03 10:33:53.332256 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round curr 0 2013-04-03 10:33:53.332258 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round new 1 2013-04-03 10:33:53.332260 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck 2013-04-03 10:33:53.332262 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck start timecheck epoch 352 round 1 2013-04-03 10:33:53.332266 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck send time_check( ping e 352 r 1 ) v1 to mon.2 10.252.24.88:6789/0 2013-04-03 10:33:53.332277 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round setting up next event 2013-04-03 10:33:53.360982 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.? 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:53.360991 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae600 from 10.252.24.88:6789/0 2013-04-03 10:33:53.361041 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:53.361047 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc6c0 from 10.252.24.88:6789/0 2013-04-03 10:33:53.361102 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:53.361108 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20c5400 from 10.252.24.88:6789/0 2013-04-03 10:33:53.362991 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:53.362998 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 2013-04-03 10:33:53.363045 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:53.363053 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae340 from 10.252.24.88:6789/0 2013-04-03 10:33:53.364593 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:53.364599 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fcd80 from 10.252.24.88:6789/0 2013-04-03 10:33:53.364739 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:53.364746 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1c18900 from 10.252.24.88:6789/0 2013-04-03 10:33:53.364828 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:53.364836 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1c186c0 from 10.252.24.88:6789/0 2013-04-03 10:33:53.372856 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 create_pending v 28958 2013-04-03 10:33:53.372874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 check_osd_map already seen 596 >= 596 2013-04-03 10:33:53.372878 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 update_logger 2013-04-03 10:33:53.372895 7fcdffcdb700 0 log [INF] : pgmap v28957: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 4872B/s wr, 0op/s 2013-04-03 10:33:53.372942 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 create_pending e185 2013-04-03 10:33:53.372961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby 2013-04-03 10:33:53.372972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 adding 10.252.58.31:6800/10908 mds.-1.0 up:standby to last_beacon 2013-04-03 10:33:53.372977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 adding 10.252.24.88:6800/17213 mds.-1.0 up:standby to last_beacon 2013-04-03 10:33:53.372980 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 adding 10.252.24.25:6802/3144 mds.0.29 up:replay to last_beacon 2013-04-03 10:33:53.372986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 update_logger 2013-04-03 10:33:53.372992 7fcdffcdb700 0 log [INF] : mdsmap e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby 2013-04-03 10:33:53.373056 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e596 create_pending e 597 2013-04-03 10:33:53.373063 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e596 remove_redundant_pg_temp 2013-04-03 10:33:53.373065 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e596 remove_down_pg_temp 2013-04-03 10:33:53.373098 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e596 update_logger 2013-04-03 10:33:53.373107 7fcdffcdb700 0 log [INF] : osdmap e596: 5 osds: 5 up, 5 in 2013-04-03 10:33:53.373129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos 2013-04-03 10:33:53.373150 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 2013-04-03 10:33:53.373181 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 create_pending v 393966 2013-04-03 10:33:53.373219 7fcdffcdb700 10 mon.hotcpc9039@0(leader).monmap v5 create_pending monmap epoch 6 2013-04-03 10:33:53.373242 7fcdffcdb700 0 log [INF] : monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} 2013-04-03 10:33:53.373264 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:53.373292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 create_pending v 2060 2013-04-03 10:33:53.373323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 AuthMonitor::on_active() 2013-04-03 10:33:53.373351 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos 2013-04-03 10:33:53.373365 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 2013-04-03 10:33:53.373375 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.373388 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:33:53.373407 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.373419 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:33:53.373428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:48.320874 mon.0 10.252.24.25:6789/0 1 : [INF] mon.hotcpc9039 calling new monitor election 2013-04-03 10:33:53.373475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(49 pgs tid 30428 v 596) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:53.373484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(49 pgs tid 30428 v 596) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:53.373489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(49 pgs tid 30428 v 596) v1 from osd.3 2013-04-03 10:33:53.373499 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.3 osd_stat(1187 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1186 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:33:53.373601 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(4 pgs tid 401 v 596) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:53.373611 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(4 pgs tid 401 v 596) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:53.373614 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(4 pgs tid 401 v 596) v1 from osd.0 2013-04-03 10:33:53.373618 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.0 osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:33:53.373655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(5 pgs tid 520 v 596) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:53.373666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(5 pgs tid 520 v 596) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:53.373669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(5 pgs tid 520 v 596) v1 from osd.2 2013-04-03 10:33:53.373673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:33:53.373706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:53.373724 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 preprocess_query auth(proto 0 36 bytes epoch 5) v1 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:33:53.373739 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 prep_auth() blob_size=36 2013-04-03 10:33:53.373816 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos 2013-04-03 10:33:53.373831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 2013-04-03 10:33:53.373839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.373851 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:33:53.373866 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 nothing new 2013-04-03 10:33:53.373895 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:boot seq 1 v0) v2 from mds.? 10.252.24.25:6802/3144 2013-04-03 10:33:53.373905 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 mds_beacon mdsbeacon(11234/hotcpc9039 up:boot seq 1 v0) v2 ignoring requested state, because mds hasn't seen latest map 2013-04-03 10:33:53.373940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(1 pgs tid 519 v 596) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:53.373947 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(1 pgs tid 519 v 596) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:53.373951 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(1 pgs tid 519 v 596) v1 from osd.2 2013-04-03 10:33:53.373954 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:33:53.373993 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e596 preprocess_query osd_boot(osd.1 booted 0 v596) v3 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:53.374004 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e596 preprocess_boot from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:53.374008 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e596 prepare_update osd_boot(osd.1 booted 0 v596) v3 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:53.374012 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e596 prepare_boot from osd.1 10.252.24.25:6805/3572 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.1 25c71ceb-f510-49a4-bf1a-ba1f96204778 e596 [95,596] lci=[593,596]) cluster_addr 10.252.24.25:6806/3572 hb_addr 10.252.24.25:6807/3572 2013-04-03 10:33:53.374022 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e596 prepare_boot was up, first marking down osd.1 10.252.24.25:6801/4077 2013-04-03 10:33:53.374028 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e596 should_propose 2013-04-03 10:33:53.374054 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(49 pgs tid 30427 v 596) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:53.374060 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(49 pgs tid 30427 v 596) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:53.374064 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(49 pgs tid 30427 v 596) v1 from osd.3 2013-04-03 10:33:53.374066 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.3 osd_stat(1187 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1186 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:33:53.374131 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 2 v184) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:33:53.374169 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(49 pgs tid 30428 v 596) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:53.374175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(49 pgs tid 30428 v 596) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:53.374179 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(49 pgs tid 30428 v 596) v1 from osd.3 2013-04-03 10:33:53.374182 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.3 osd_stat(1187 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1186 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:33:53.374241 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(4 pgs tid 401 v 596) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:53.374249 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(4 pgs tid 401 v 596) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:53.374252 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(4 pgs tid 401 v 596) v1 from osd.0 2013-04-03 10:33:53.374256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.0 osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:33:53.374286 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(5 pgs tid 520 v 596) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:53.374293 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(5 pgs tid 520 v 596) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:53.374296 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(5 pgs tid 520 v 596) v1 from osd.2 2013-04-03 10:33:53.374299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:33:53.374331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos 2013-04-03 10:33:53.374346 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 2013-04-03 10:33:53.374362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:53.374394 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck time_check( pong e 352 r 1 ts 2013-04-03 10:33:53.371770 ) v1 2013-04-03 10:33:53.374401 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader time_check( pong e 352 r 1 ts 2013-04-03 10:33:53.371770 ) v1 2013-04-03 10:33:53.374408 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader from mon.2 10.252.24.88:6789/0 ts 2013-04-03 10:33:53.371770 delta -0.002635 skew_bound 0 latency 0.04214 2013-04-03 10:33:53.374422 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader got pongs from everybody (2 total) 2013-04-03 10:33:53.374425 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish_round curr 1 2013-04-03 10:33:53.374427 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_report 2013-04-03 10:33:53.374430 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_report send report to mon.2 10.252.24.88:6789/0 2013-04-03 10:33:53.374464 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos 2013-04-03 10:33:53.374483 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 2013-04-03 10:33:53.374492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.374503 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:33:53.374526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.374537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:33:53.374546 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:48.320874 mon.0 10.252.24.25:6789/0 1 : [INF] mon.hotcpc9039 calling new monitor election 2013-04-03 10:33:53.374565 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:53.331862 mon.0 10.252.24.25:6789/0 2 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 2013-04-03 10:33:53.374626 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos 2013-04-03 10:33:53.374640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 2013-04-03 10:33:53.374655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:53.374692 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos 2013-04-03 10:33:53.374705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 2013-04-03 10:33:53.374712 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.374722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:33:53.374745 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.374756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:33:53.374764 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:53.331862 mon.0 10.252.24.25:6789/0 2 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 2013-04-03 10:33:53.374784 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:53.372898 mon.0 10.252.24.25:6789/0 3 : [INF] pgmap v28957: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 4872B/s wr, 0op/s 2013-04-03 10:33:53.374836 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos 2013-04-03 10:33:53.374850 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 2013-04-03 10:33:53.374858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.374869 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:33:53.374891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.374902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:33:53.374910 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:53.372898 mon.0 10.252.24.25:6789/0 3 : [INF] pgmap v28957: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 4872B/s wr, 0op/s 2013-04-03 10:33:53.374929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:53.372993 mon.0 10.252.24.25:6789/0 4 : [INF] mdsmap e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby 2013-04-03 10:33:53.374977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos 2013-04-03 10:33:53.374990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 2013-04-03 10:33:53.374998 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.375009 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:33:53.375032 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.375042 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:33:53.375051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:53.372993 mon.0 10.252.24.25:6789/0 4 : [INF] mdsmap e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby 2013-04-03 10:33:53.375069 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:53.373108 mon.0 10.252.24.25:6789/0 5 : [INF] osdmap e596: 5 osds: 5 up, 5 in 2013-04-03 10:33:53.375124 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:53.375139 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 preprocess_query auth(proto 2 144 bytes epoch 0) v1 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:33:53.375150 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 prep_auth() blob_size=144 2013-04-03 10:33:53.375912 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=185+,monmap=6+}) v2 2013-04-03 10:33:53.375921 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:33:53.381973 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:53.381980 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x25f0840 from 10.252.24.88:6789/0 2013-04-03 10:33:53.382008 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 3 v184) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:33:53.423616 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393965 encode_pending v393966 2013-04-03 10:33:53.424033 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28957 encode_pending v 28958 2013-04-03 10:33:53.424265 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e596 encode_pending e 597 2013-04-03 10:33:53.424273 7fce004dc700 2 mon.hotcpc9039@0(leader).osd e596 osd.1 DOWN 2013-04-03 10:33:53.458440 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos 2013-04-03 10:33:53.458464 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos version 393966 summary v 393965 2013-04-03 10:33:53.458478 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos latest full 393798 2013-04-03 10:33:53.458496 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 update_from_paxos applying incremental log 393966 2013-04-03 10:33:48.320874 mon.0 10.252.24.25:6789/0 1 : [INF] mon.hotcpc9039 calling new monitor election 2013-04-03 10:33:53.458521 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 update_from_paxos applying incremental log 393966 2013-04-03 10:33:53.331862 mon.0 10.252.24.25:6789/0 2 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 2013-04-03 10:33:53.458541 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 update_from_paxos applying incremental log 393966 2013-04-03 10:33:53.372898 mon.0 10.252.24.25:6789/0 3 : [INF] pgmap v28957: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 4872B/s wr, 0op/s 2013-04-03 10:33:53.458560 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 update_from_paxos applying incremental log 393966 2013-04-03 10:33:53.372993 mon.0 10.252.24.25:6789/0 4 : [INF] mdsmap e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby 2013-04-03 10:33:53.458578 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 update_from_paxos applying incremental log 393966 2013-04-03 10:33:53.373108 mon.0 10.252.24.25:6789/0 5 : [INF] osdmap e596: 5 osds: 5 up, 5 in 2013-04-03 10:33:53.458611 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 check_subs 2013-04-03 10:33:53.458635 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 create_pending v 393967 2013-04-03 10:33:53.458655 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.458675 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.458693 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.458709 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.458724 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.494454 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28957 update_from_paxos applying incremental 28958 2013-04-03 10:33:53.494627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 v28958: 240 pgs: 240 active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 1302B/s wr, 13op/s 2013-04-03 10:33:53.494661 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 send_pg_creates to 0 pgs 2013-04-03 10:33:53.494666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 update_logger 2013-04-03 10:33:53.494684 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 create_pending v 28959 2013-04-03 10:33:53.494696 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:53.494720 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:53.494757 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:53.494784 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:53.494811 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:53.494854 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:53.494887 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:53.494918 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:53.494945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 check_osd_map already seen 596 >= 596 2013-04-03 10:33:53.494949 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 update_logger 2013-04-03 10:33:53.494959 7fcdffcdb700 0 log [INF] : pgmap v28958: 240 pgs: 240 active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 1302B/s wr, 13op/s 2013-04-03 10:33:53.495679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos 2013-04-03 10:33:53.495694 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos version 393966 summary v 393966 2013-04-03 10:33:53.495702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.495713 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:33:53.495738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.495749 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:33:53.495759 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 logging 2013-04-03 10:33:53.373245 mon.0 10.252.24.25:6789/0 6 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} 2013-04-03 10:33:53.495779 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 logging 2013-04-03 10:33:53.494960 mon.0 10.252.24.25:6789/0 7 : [INF] pgmap v28958: 240 pgs: 240 active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 1302B/s wr, 13op/s 2013-04-03 10:33:53.540285 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e596 update_from_paxos applying incremental 597 2013-04-03 10:33:53.540337 7fcdffcdb700 1 mon.hotcpc9039@0(leader).osd e597 e597: 5 osds: 4 up, 5 in 2013-04-03 10:33:53.541500 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 adding osd.1 to down_pending_out map 2013-04-03 10:33:53.541556 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 check_osd_map applying osdmap e597 to pg_map 2013-04-03 10:33:53.541576 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 register_new_pgs checking pg pools for osdmap epoch 597, last_pg_scan 341 2013-04-03 10:33:53.541580 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 2013-04-03 10:33:53.541585 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:33:53.541588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:33:53.541590 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 2013-04-03 10:33:53.541593 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 2013-04-03 10:33:53.541596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 2013-04-03 10:33:53.541598 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 2013-04-03 10:33:53.541601 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 2013-04-03 10:33:53.541603 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 2013-04-03 10:33:53.541619 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs 2013-04-03 10:33:53.541621 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 check_down_pgs 2013-04-03 10:33:53.541623 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.b stale with acting [1,0] 2013-04-03 10:33:53.541626 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 7.6 stale with acting [1,0] 2013-04-03 10:33:53.541628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 6.7 stale with acting [1,0] 2013-04-03 10:33:53.541632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.3f stale with acting [1,0] 2013-04-03 10:33:53.541634 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.3f stale with acting [1,3] 2013-04-03 10:33:53.541636 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.3c stale with acting [1,3] 2013-04-03 10:33:53.541639 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.3d stale with acting [1,3] 2013-04-03 10:33:53.541641 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.3e stale with acting [1,0] 2013-04-03 10:33:53.541643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.3b stale with acting [1,3] 2013-04-03 10:33:53.541646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.38 stale with acting [1,4] 2013-04-03 10:33:53.541648 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.39 stale with acting [1,4] 2013-04-03 10:33:53.541650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.37 stale with acting [1,4] 2013-04-03 10:33:53.541652 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.34 stale with acting [1,3] 2013-04-03 10:33:53.541654 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.35 stale with acting [1,3] 2013-04-03 10:33:53.541656 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.34 stale with acting [1,2] 2013-04-03 10:33:53.541658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.33 stale with acting [1,2] 2013-04-03 10:33:53.541660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.33 stale with acting [1,3] 2013-04-03 10:33:53.541663 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.30 stale with acting [1,2] 2013-04-03 10:33:53.541665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.31 stale with acting [1,2] 2013-04-03 10:33:53.541667 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.32 stale with acting [1,2] 2013-04-03 10:33:53.541669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.30 stale with acting [1,0] 2013-04-03 10:33:53.541671 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.2d stale with acting [1,4] 2013-04-03 10:33:53.541673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.2e stale with acting [1,4] 2013-04-03 10:33:53.541675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.2f stale with acting [1,4] 2013-04-03 10:33:53.541677 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.2c stale with acting [1,2] 2013-04-03 10:33:53.541680 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.2f stale with acting [1,0] 2013-04-03 10:33:53.541683 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.2e stale with acting [1,2] 2013-04-03 10:33:53.541685 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.2f stale with acting [1,2] 2013-04-03 10:33:53.541688 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.2e stale with acting [1,0] 2013-04-03 10:33:53.541690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.2d stale with acting [1,2] 2013-04-03 10:33:53.541692 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.25 stale with acting [1,2] 2013-04-03 10:33:53.541694 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.26 stale with acting [1,2] 2013-04-03 10:33:53.541696 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.27 stale with acting [1,2] 2013-04-03 10:33:53.541699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.24 stale with acting [1,3] 2013-04-03 10:33:53.541701 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.26 stale with acting [1,3] 2013-04-03 10:33:53.541703 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.25 stale with acting [1,3] 2013-04-03 10:33:53.541705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.20 stale with acting [1,2] 2013-04-03 10:33:53.541707 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.1d stale with acting [1,2] 2013-04-03 10:33:53.541709 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.1e stale with acting [1,2] 2013-04-03 10:33:53.541712 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.1f stale with acting [1,2] 2013-04-03 10:33:53.541714 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.1f stale with acting [1,2] 2013-04-03 10:33:53.541716 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.1e stale with acting [1,2] 2013-04-03 10:33:53.541719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.d stale with acting [1,0] 2013-04-03 10:33:53.541722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.c stale with acting [1,0] 2013-04-03 10:33:53.541724 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 8.5 stale with acting [1,0] 2013-04-03 10:33:53.541770 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 encode_pending v 28959 2013-04-03 10:33:53.544157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 send_pg_creates to 0 pgs 2013-04-03 10:33:53.544163 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 send_to_waiting 597 2013-04-03 10:33:53.544167 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 share_map_with_random_osd no up osd on our session map 2013-04-03 10:33:53.544169 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 update_logger 2013-04-03 10:33:53.544188 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 create_pending e 598 2013-04-03 10:33:53.544190 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 remove_redundant_pg_temp 2013-04-03 10:33:53.544191 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 remove_down_pg_temp 2013-04-03 10:33:53.544227 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_query osd_boot(osd.1 booted 0 v596) v3 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:53.544236 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_boot from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:53.544239 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_update osd_boot(osd.1 booted 0 v596) v3 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:53.544242 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_boot from osd.1 10.252.24.25:6805/3572 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.1 25c71ceb-f510-49a4-bf1a-ba1f96204778 e596 [95,596] lci=[593,596]) cluster_addr 10.252.24.25:6806/3572 hb_addr 10.252.24.25:6807/3572 2013-04-03 10:33:53.544254 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 setting osd.1 uuid to 25c71ceb-f510-49a4-bf1a-ba1f96204778 2013-04-03 10:33:53.544257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 old osd_info: up_from 593 up_thru 593 down_at 597 last_clean_interval [582,590) 2013-04-03 10:33:53.544259 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 prepare_boot osd.1 last_clean_interval [582,590) -> [593-596) 2013-04-03 10:33:53.544263 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 not laggy, new xi down_stamp 2013-04-03 10:33:53.424273 laggy_probability 0 laggy_interval 0 2013-04-03 10:33:53.544269 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 should_propose 2013-04-03 10:33:53.544281 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 update_logger 2013-04-03 10:33:53.544286 7fcdffcdb700 0 log [INF] : osdmap e597: 5 osds: 4 up, 5 in 2013-04-03 10:33:53.544340 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos 2013-04-03 10:33:53.544354 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos version 393966 summary v 393966 2013-04-03 10:33:53.544363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.544373 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:33:53.544391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 nothing new 2013-04-03 10:33:53.603818 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 update_from_paxos applying incremental 28959 2013-04-03 10:33:53.603938 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 v28959: 240 pgs: 195 active+clean, 45 stale+active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 0B/s wr, 15op/s 2013-04-03 10:33:53.603968 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 send_pg_creates to 0 pgs 2013-04-03 10:33:53.603972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 update_logger 2013-04-03 10:33:53.603985 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 create_pending v 28960 2013-04-03 10:33:53.603994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 check_osd_map already seen 597 >= 597 2013-04-03 10:33:53.603997 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 update_logger 2013-04-03 10:33:53.604008 7fcdffcdb700 0 log [INF] : pgmap v28959: 240 pgs: 195 active+clean, 45 stale+active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 0B/s wr, 15op/s 2013-04-03 10:33:53.604063 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos 2013-04-03 10:33:53.604078 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos version 393966 summary v 393966 2013-04-03 10:33:53.604094 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:53.604134 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos 2013-04-03 10:33:53.604148 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos version 393966 summary v 393966 2013-04-03 10:33:53.604157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.604169 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:33:53.604192 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:53.604204 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:33:53.604212 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 logging 2013-04-03 10:33:53.494960 mon.0 10.252.24.25:6789/0 7 : [INF] pgmap v28958: 240 pgs: 240 active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 1302B/s wr, 13op/s 2013-04-03 10:33:53.604232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 logging 2013-04-03 10:33:53.544287 mon.0 10.252.24.25:6789/0 8 : [INF] osdmap e597: 5 osds: 4 up, 5 in 2013-04-03 10:33:53.768211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos 2013-04-03 10:33:53.768227 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos version 393966 summary v 393966 2013-04-03 10:33:53.768241 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:53.768266 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:53.768271 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2294a80 from 10.252.24.88:6789/0 2013-04-03 10:33:53.768292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_query osd_alive(want up_thru 597 have 597) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:53.768297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_alive want up_thru 597 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:53.768301 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_update osd_alive(want up_thru 597 have 597) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:53.768306 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_alive want up_thru 597 have 597 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:53.768310 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 should_propose 2013-04-03 10:33:53.770072 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:53.770080 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1c18900 from 10.252.24.88:6789/0 2013-04-03 10:33:53.770104 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 preprocess_query pg_stats(30 pgs tid 30429 v 597) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:53.770111 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 prepare_update pg_stats(30 pgs tid 30429 v 597) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:53.770115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 prepare_pg_stats pg_stats(30 pgs tid 30429 v 597) v1 from osd.3 2013-04-03 10:33:53.770119 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 got osd.3 osd_stat(1187 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1187 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:33:53.770174 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:53.770181 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2294c40 from 10.252.24.88:6789/0 2013-04-03 10:33:53.770203 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_query osd_alive(want up_thru 597 have 597) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:53.770209 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_alive want up_thru 597 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:53.770212 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_update osd_alive(want up_thru 597 have 597) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:53.770215 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_alive want up_thru 597 have 597 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:53.770218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 should_propose 2013-04-03 10:33:54.351383 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:54.351393 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2294e00 from 10.252.24.88:6789/0 2013-04-03 10:33:54.351440 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_query osd_alive(want up_thru 597 have 597) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:33:54.351448 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_alive want up_thru 597 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:33:54.351452 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_update osd_alive(want up_thru 597 have 597) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:33:54.351458 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_alive want up_thru 597 have 597 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:33:54.351462 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 should_propose 2013-04-03 10:33:54.413641 7fcdfead7700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.88:6800/17213 mds protocol 0 2013-04-03 10:33:54.413997 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:33:54.414004 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: mds.? 10.252.24.88:6800/17213 is open for mds.? 10.252.24.88:6800/17213 2013-04-03 10:33:54.414018 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:33:54.414043 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:54.414059 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 preprocess_query auth(proto 0 36 bytes epoch 5) v1 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:33:54.414071 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 prep_auth() blob_size=36 2013-04-03 10:33:54.444079 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:54.444096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 preprocess_query auth(proto 2 144 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:33:54.444107 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 prep_auth() blob_size=144 2013-04-03 10:33:54.445593 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=185+,monmap=6+}) v2 2013-04-03 10:33:54.445608 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:33:54.493440 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393966 encode_pending v393967 2013-04-03 10:33:54.527132 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos 2013-04-03 10:33:54.527155 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393966 2013-04-03 10:33:54.527168 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos latest full 393798 2013-04-03 10:33:54.527186 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393967 update_from_paxos applying incremental log 393967 2013-04-03 10:33:53.373245 mon.0 10.252.24.25:6789/0 6 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} 2013-04-03 10:33:54.527213 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393967 update_from_paxos applying incremental log 393967 2013-04-03 10:33:53.494960 mon.0 10.252.24.25:6789/0 7 : [INF] pgmap v28958: 240 pgs: 240 active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 1302B/s wr, 13op/s 2013-04-03 10:33:54.527234 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393967 update_from_paxos applying incremental log 393967 2013-04-03 10:33:53.544287 mon.0 10.252.24.25:6789/0 8 : [INF] osdmap e597: 5 osds: 4 up, 5 in 2013-04-03 10:33:54.527268 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 check_subs 2013-04-03 10:33:54.527295 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 create_pending v 393968 2013-04-03 10:33:54.527316 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393967 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:33:54.527335 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393967 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:33:54.527388 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos 2013-04-03 10:33:54.527405 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393967 2013-04-03 10:33:54.527421 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:54.538205 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos 2013-04-03 10:33:54.538221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393967 2013-04-03 10:33:54.538237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:54.540048 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e597 min_last_epoch_clean 593 2013-04-03 10:33:54.540108 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e597 min_last_epoch_clean 593 2013-04-03 10:33:54.540162 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e597 encode_pending e 598 2013-04-03 10:33:54.540166 7fce004dc700 2 mon.hotcpc9039@0(leader).osd e597 osd.1 UP 10.252.24.25:6805/3572 2013-04-03 10:33:54.558575 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 update_from_paxos applying incremental 598 2013-04-03 10:33:54.558625 7fcdffcdb700 1 mon.hotcpc9039@0(leader).osd e598 e598: 5 osds: 5 up, 5 in 2013-04-03 10:33:54.559927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 check_osd_map applying osdmap e598 to pg_map 2013-04-03 10:33:54.559952 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 register_new_pgs checking pg pools for osdmap epoch 598, last_pg_scan 341 2013-04-03 10:33:54.559957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 2013-04-03 10:33:54.559961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:33:54.559963 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:33:54.559966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 2013-04-03 10:33:54.559968 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 2013-04-03 10:33:54.559971 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 2013-04-03 10:33:54.559973 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 2013-04-03 10:33:54.559975 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 2013-04-03 10:33:54.559979 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 2013-04-03 10:33:54.559988 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs 2013-04-03 10:33:54.559990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 check_down_pgs 2013-04-03 10:33:54.560039 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 encode_pending v 28960 2013-04-03 10:33:54.561199 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 send_pg_creates to 0 pgs 2013-04-03 10:33:54.561203 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 send_to_waiting 598 2013-04-03 10:33:54.561206 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 share_map_with_random_osd no up osd on our session map 2013-04-03 10:33:54.561209 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 update_logger 2013-04-03 10:33:54.561228 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 create_pending e 599 2013-04-03 10:33:54.561230 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 remove_redundant_pg_temp 2013-04-03 10:33:54.561232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 remove_down_pg_temp 2013-04-03 10:33:54.561251 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 _booted osd.1 10.252.24.25:6805/3572 w 0 from 596 2013-04-03 10:33:54.561268 7fcdffcdb700 0 log [INF] : osd.1 10.252.24.25:6805/3572 boot 2013-04-03 10:33:54.561301 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_latest to osd.1 10.252.24.25:6805/3572 start 597 2013-04-03 10:33:54.561307 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_incremental [597..598] to osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:54.561331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 build_incremental [597..598] 2013-04-03 10:33:54.561382 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 _reply_map 597 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:54.561402 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_latest to osd.0 10.252.58.31:6802/2682 start 597 2013-04-03 10:33:54.561407 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_incremental [597..598] to osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:54.561417 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 build_incremental [597..598] 2013-04-03 10:33:54.561456 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 _reply_map 597 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:54.561474 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_latest to osd.3 10.252.24.88:6801/17538 start 597 2013-04-03 10:33:54.561479 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_incremental [597..598] to osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:54.561489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 build_incremental [597..598] 2013-04-03 10:33:54.561529 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 _reply_map 597 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:33:54.561547 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_latest to osd.4 10.252.24.32:6800/13473 start 597 2013-04-03 10:33:54.561552 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_incremental [597..598] to osd.4 10.252.24.32:6800/13473 2013-04-03 10:33:54.561568 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 build_incremental [597..598] 2013-04-03 10:33:54.561607 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 update_logger 2013-04-03 10:33:54.561613 7fcdffcdb700 0 log [INF] : osdmap e598: 5 osds: 5 up, 5 in 2013-04-03 10:33:54.561665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos 2013-04-03 10:33:54.561680 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393967 2013-04-03 10:33:54.561689 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:54.561699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:33:54.561723 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:54.561735 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:33:54.561744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 logging 2013-04-03 10:33:53.604009 mon.0 10.252.24.25:6789/0 9 : [INF] pgmap v28959: 240 pgs: 195 active+clean, 45 stale+active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 0B/s wr, 15op/s 2013-04-03 10:33:54.561765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 logging 2013-04-03 10:33:54.561269 mon.0 10.252.24.25:6789/0 10 : [INF] osd.1 10.252.24.25:6805/3572 boot 2013-04-03 10:33:54.561815 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos 2013-04-03 10:33:54.561831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393967 2013-04-03 10:33:54.561840 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:54.561850 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:33:54.561866 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 nothing new 2013-04-03 10:33:54.672946 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28959 update_from_paxos applying incremental 28960 2013-04-03 10:33:54.673036 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 v28960: 240 pgs: 190 active+clean, 38 stale+active+clean, 12 peering; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:33:54.673062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 send_pg_creates to 0 pgs 2013-04-03 10:33:54.673065 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 update_logger 2013-04-03 10:33:54.673077 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 create_pending v 28961 2013-04-03 10:33:54.673085 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28960 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:54.673106 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 check_osd_map already seen 598 >= 598 2013-04-03 10:33:54.673110 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 update_logger 2013-04-03 10:33:54.673121 7fcdffcdb700 0 log [INF] : pgmap v28960: 240 pgs: 190 active+clean, 38 stale+active+clean, 12 peering; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:33:54.673179 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos 2013-04-03 10:33:54.673195 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393967 2013-04-03 10:33:54.673212 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:54.673251 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos 2013-04-03 10:33:54.673266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393967 2013-04-03 10:33:54.673274 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:54.673286 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:33:54.673318 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:54.673332 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:33:54.673343 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 logging 2013-04-03 10:33:54.561269 mon.0 10.252.24.25:6789/0 10 : [INF] osd.1 10.252.24.25:6805/3572 boot 2013-04-03 10:33:54.673362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 logging 2013-04-03 10:33:54.561614 mon.0 10.252.24.25:6789/0 11 : [INF] osdmap e598: 5 osds: 5 up, 5 in 2013-04-03 10:33:54.892852 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos 2013-04-03 10:33:54.892870 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393967 2013-04-03 10:33:54.892888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:54.892918 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:54.892926 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x22248c0 from 10.252.24.88:6789/0 2013-04-03 10:33:54.892949 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_query osd_alive(want up_thru 598 have 598) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:54.892956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_alive want up_thru 598 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:54.892960 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_update osd_alive(want up_thru 598 have 598) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:54.892964 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_alive want up_thru 598 have 598 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:54.892967 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 should_propose 2013-04-03 10:33:54.893187 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:54.893194 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x22941c0 from 10.252.24.88:6789/0 2013-04-03 10:33:54.893216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_query osd_alive(want up_thru 598 have 598) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:54.893223 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_alive want up_thru 598 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:54.893226 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_update osd_alive(want up_thru 598 have 598) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:54.893229 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_alive want up_thru 598 have 598 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:54.893232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 should_propose 2013-04-03 10:33:54.893249 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:54.893255 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2294380 from 10.252.24.88:6789/0 2013-04-03 10:33:54.893275 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_query osd_alive(want up_thru 598 have 598) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:54.893281 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_alive want up_thru 598 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:54.893286 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_update osd_alive(want up_thru 598 have 598) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:54.893289 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_alive want up_thru 598 have 598 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:54.893291 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 should_propose 2013-04-03 10:33:54.893462 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:54.893468 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae340 from 10.252.24.88:6789/0 2013-04-03 10:33:54.893491 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 4 v184) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:33:54.936884 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:54.936891 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2294540 from 10.252.24.88:6789/0 2013-04-03 10:33:54.936923 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_query osd_alive(want up_thru 598 have 598) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:54.936930 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_alive want up_thru 598 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:54.936933 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_update osd_alive(want up_thru 598 have 598) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:54.936936 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_alive want up_thru 598 have 598 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:54.936939 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 should_propose 2013-04-03 10:33:55.550269 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:55.550278 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2224380 from 10.252.24.88:6789/0 2013-04-03 10:33:55.550326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_query osd_alive(want up_thru 598 have 598) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:55.550334 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_alive want up_thru 598 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:55.550338 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_update osd_alive(want up_thru 598 have 598) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:55.550341 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_alive want up_thru 598 have 598 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:55.550345 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 should_propose 2013-04-03 10:33:55.551510 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393967 encode_pending v393968 2013-04-03 10:33:55.553941 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:55.553948 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1c18900 from 10.252.24.88:6789/0 2013-04-03 10:33:55.553975 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 preprocess_query pg_stats(45 pgs tid 1 v 598) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:55.553982 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 prepare_update pg_stats(45 pgs tid 1 v 598) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:55.553986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 prepare_pg_stats pg_stats(45 pgs tid 1 v 598) v1 from osd.1 2013-04-03 10:33:55.553992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 got osd.1 osd_stat(1110 MB used, 148 GB avail, 149 GB total, peers []/[]) (was osd_stat(1112 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:33:55.575996 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos 2013-04-03 10:33:55.576022 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos version 393968 summary v 393967 2013-04-03 10:33:55.576036 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos latest full 393798 2013-04-03 10:33:55.576053 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393968 update_from_paxos applying incremental log 393968 2013-04-03 10:33:53.604009 mon.0 10.252.24.25:6789/0 9 : [INF] pgmap v28959: 240 pgs: 195 active+clean, 45 stale+active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 0B/s wr, 15op/s 2013-04-03 10:33:55.576077 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393968 update_from_paxos applying incremental log 393968 2013-04-03 10:33:54.561269 mon.0 10.252.24.25:6789/0 10 : [INF] osd.1 10.252.24.25:6805/3572 boot 2013-04-03 10:33:55.576096 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393968 update_from_paxos applying incremental log 393968 2013-04-03 10:33:54.561614 mon.0 10.252.24.25:6789/0 11 : [INF] osdmap e598: 5 osds: 5 up, 5 in 2013-04-03 10:33:55.576135 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 check_subs 2013-04-03 10:33:55.576162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 create_pending v 393969 2013-04-03 10:33:55.576183 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393968 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:33:55.576203 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393968 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:33:55.576255 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos 2013-04-03 10:33:55.576269 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos version 393968 summary v 393968 2013-04-03 10:33:55.576286 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:55.576370 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140292 v184) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:33:55.588205 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos 2013-04-03 10:33:55.588221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos version 393968 summary v 393968 2013-04-03 10:33:55.588237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:55.634223 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e598 min_last_epoch_clean 593 2013-04-03 10:33:55.634292 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e598 min_last_epoch_clean 593 2013-04-03 10:33:55.634341 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e598 encode_pending e 599 2013-04-03 10:33:55.634946 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28960 encode_pending v 28961 2013-04-03 10:33:55.650533 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 update_from_paxos applying incremental 599 2013-04-03 10:33:55.650576 7fcdffcdb700 1 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in 2013-04-03 10:33:55.651761 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 check_osd_map -- pgmap not writeable, waiting 2013-04-03 10:33:55.651769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 send_to_waiting 599 2013-04-03 10:33:55.651773 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 share_map_with_random_osd no up osd on our session map 2013-04-03 10:33:55.651774 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 update_logger 2013-04-03 10:33:55.651790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 create_pending e 600 2013-04-03 10:33:55.651793 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 remove_redundant_pg_temp 2013-04-03 10:33:55.651794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 remove_down_pg_temp 2013-04-03 10:33:55.651813 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e599 _reply_map 598 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:55.651831 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_latest to osd.1 10.252.24.25:6805/3572 start 598 2013-04-03 10:33:55.651834 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_incremental [598..599] to osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:55.651845 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 build_incremental [598..599] 2013-04-03 10:33:55.651890 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e599 _reply_map 598 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:55.651911 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_latest to osd.2 10.252.24.30:6801/10373 start 598 2013-04-03 10:33:55.651916 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_incremental [598..599] to osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:55.651927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 build_incremental [598..599] 2013-04-03 10:33:55.651967 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e599 _reply_map 598 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:55.651985 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_latest to osd.0 10.252.58.31:6802/2682 start 598 2013-04-03 10:33:55.651990 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_incremental [598..599] to osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:55.652000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 build_incremental [598..599] 2013-04-03 10:33:55.652038 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e599 _reply_map 598 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:55.652057 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_latest to osd.3 10.252.24.88:6801/17538 start 598 2013-04-03 10:33:55.652062 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_incremental [598..599] to osd.3 10.252.24.88:6801/17538 2013-04-03 10:33:55.652072 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 build_incremental [598..599] 2013-04-03 10:33:55.652109 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e599 _reply_map 598 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:55.652132 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_latest to osd.1 10.252.24.25:6805/3572 start 598 2013-04-03 10:33:55.652138 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_incremental [598..599] to osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:55.652148 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 build_incremental [598..599] 2013-04-03 10:33:55.652190 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 update_logger 2013-04-03 10:33:55.652199 7fcdffcdb700 0 log [INF] : osdmap e599: 5 osds: 5 up, 5 in 2013-04-03 10:33:55.653484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos 2013-04-03 10:33:55.653499 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos version 393968 summary v 393968 2013-04-03 10:33:55.653508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:55.653520 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:33:55.653545 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:55.653557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:33:55.653566 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 logging 2013-04-03 10:33:54.673122 mon.0 10.252.24.25:6789/0 12 : [INF] pgmap v28960: 240 pgs: 190 active+clean, 38 stale+active+clean, 12 peering; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:33:55.653585 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 logging 2013-04-03 10:33:55.652200 mon.0 10.252.24.25:6789/0 13 : [INF] osdmap e599: 5 osds: 5 up, 5 in 2013-04-03 10:33:55.977494 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28960 update_from_paxos applying incremental 28961 2013-04-03 10:33:55.977623 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 v28961: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:33:55.977650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 send_pg_creates to 0 pgs 2013-04-03 10:33:55.977655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 update_logger 2013-04-03 10:33:55.977672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 create_pending v 28962 2013-04-03 10:33:55.977682 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28961 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:33:55.977730 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 check_osd_map applying osdmap e599 to pg_map 2013-04-03 10:33:55.977752 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 register_new_pgs checking pg pools for osdmap epoch 599, last_pg_scan 341 2013-04-03 10:33:55.977756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 2013-04-03 10:33:55.977760 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:33:55.977763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:33:55.977765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 2013-04-03 10:33:55.977768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 2013-04-03 10:33:55.977770 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 2013-04-03 10:33:55.977773 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 2013-04-03 10:33:55.977777 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 2013-04-03 10:33:55.977779 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 2013-04-03 10:33:55.977792 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs 2013-04-03 10:33:55.977794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 check_down_pgs 2013-04-03 10:33:55.977845 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 encode_pending v 28962 2013-04-03 10:33:56.014058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 send_pg_creates to 0 pgs 2013-04-03 10:33:56.135870 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28961 update_from_paxos applying incremental 28962 2013-04-03 10:33:56.135880 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 v28962: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:33:56.135897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 send_pg_creates to 0 pgs 2013-04-03 10:33:56.135900 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 update_logger 2013-04-03 10:33:56.135908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 create_pending v 28963 2013-04-03 10:33:56.135916 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 check_osd_map already seen 599 >= 599 2013-04-03 10:33:56.135920 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 update_logger 2013-04-03 10:33:56.135932 7fcdffcdb700 0 log [INF] : pgmap v28962: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:33:56.135987 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos 2013-04-03 10:33:56.136006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos version 393968 summary v 393968 2013-04-03 10:33:56.136025 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:56.136065 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos 2013-04-03 10:33:56.136080 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos version 393968 summary v 393968 2013-04-03 10:33:56.136089 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:56.136099 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:33:56.136120 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 nothing new 2013-04-03 10:33:56.163044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos 2013-04-03 10:33:56.163063 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos version 393968 summary v 393968 2013-04-03 10:33:56.163078 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:56.650245 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393968 encode_pending v393969 2013-04-03 10:33:56.696016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos 2013-04-03 10:33:56.696040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393968 2013-04-03 10:33:56.696054 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos latest full 393798 2013-04-03 10:33:56.696073 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393969 update_from_paxos applying incremental log 393969 2013-04-03 10:33:54.673122 mon.0 10.252.24.25:6789/0 12 : [INF] pgmap v28960: 240 pgs: 190 active+clean, 38 stale+active+clean, 12 peering; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:33:56.696099 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393969 update_from_paxos applying incremental log 393969 2013-04-03 10:33:55.652200 mon.0 10.252.24.25:6789/0 13 : [INF] osdmap e599: 5 osds: 5 up, 5 in 2013-04-03 10:33:56.696137 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 check_subs 2013-04-03 10:33:56.696165 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 create_pending v 393970 2013-04-03 10:33:56.696186 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393969 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:33:56.696245 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos 2013-04-03 10:33:56.696260 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 2013-04-03 10:33:56.696277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:56.733389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos 2013-04-03 10:33:56.733405 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 2013-04-03 10:33:56.733419 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:57.375769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21663 v184) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:33:58.318478 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28962 check_down_pgs 2013-04-03 10:33:58.318505 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28962 v28962: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:33:58.318535 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e184 e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby 2013-04-03 10:33:58.318560 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in 2013-04-03 10:33:58.318596 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 2013-04-03 10:33:58.318625 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos 2013-04-03 10:33:58.318642 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 2013-04-03 10:33:58.318651 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393969 log 2013-04-03 10:33:58.318667 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos 2013-04-03 10:33:58.318684 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2059 auth 2013-04-03 10:33:58.318703 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2059 AuthMonitor::tick() updated rotating, now calling propose_pending 2013-04-03 10:33:58.318740 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2059 encode_full auth v 2059 2013-04-03 10:33:58.318827 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2059 encode_pending v 2060 2013-04-03 10:33:58.377413 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:33:58.377448 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos version 2060 keys ver 2059 latest 2059 2013-04-03 10:33:58.377461 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos key server version 2059 2013-04-03 10:33:58.377493 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos() last_allocated_id=11896 max_global_id=11896 2013-04-03 10:33:58.377511 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 create_pending v 2061 2013-04-03 10:33:58.377532 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 AuthMonitor::on_active() 2013-04-03 10:33:58.377575 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos 2013-04-03 10:33:58.377591 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 2013-04-03 10:33:58.377606 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:33:58.417310 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos 2013-04-03 10:33:58.417333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 2013-04-03 10:33:58.417347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:33:58.754745 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:58.754754 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e3b00 from 10.252.24.88:6789/0 2013-04-03 10:33:58.754787 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 5 v184) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:33:59.399627 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:59.399637 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc480 from 10.252.24.88:6789/0 2013-04-03 10:33:59.399686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 preprocess_query pg_stats(0 pgs tid 56 v 599) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:33:59.399696 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 prepare_update pg_stats(0 pgs tid 56 v 599) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:33:59.399699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 prepare_pg_stats pg_stats(0 pgs tid 56 v 599) v1 from osd.4 2013-04-03 10:33:59.399706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 got osd.4 osd_stat(1052 MB used, 56605 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1052 MB used, 56605 MB avail, 59728 MB total, peers []/[])) 2013-04-03 10:33:59.449885 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28962 encode_pending v 28963 2013-04-03 10:33:59.492578 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28962 update_from_paxos applying incremental 28963 2013-04-03 10:33:59.492587 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 v28963: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:33:59.492604 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 send_pg_creates to 0 pgs 2013-04-03 10:33:59.492608 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 update_logger 2013-04-03 10:33:59.492617 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 create_pending v 28964 2013-04-03 10:33:59.492626 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28963 _updated_stats for osd.4 10.252.24.32:6800/13473 2013-04-03 10:33:59.492641 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 check_osd_map already seen 599 >= 599 2013-04-03 10:33:59.492644 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 update_logger 2013-04-03 10:33:59.493016 7fcdffcdb700 0 log [INF] : pgmap v28963: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:33:59.493400 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos 2013-04-03 10:33:59.493415 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 2013-04-03 10:33:59.493424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:59.493436 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:33:59.493463 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:33:59.493475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:33:59.493486 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 logging 2013-04-03 10:33:56.135933 mon.0 10.252.24.25:6789/0 14 : [INF] pgmap v28962: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:33:59.493507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 logging 2013-04-03 10:33:59.493020 mon.0 10.252.24.25:6789/0 15 : [INF] pgmap v28963: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:33:59.518593 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos 2013-04-03 10:33:59.518611 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 2013-04-03 10:33:59.518627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:33:59.544361 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos 2013-04-03 10:33:59.544377 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 2013-04-03 10:33:59.544391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:33:59.575445 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140293 v184) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:33:59.785930 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:59.785939 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc6c0 from 10.252.24.88:6789/0 2013-04-03 10:33:59.785988 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 preprocess_query pg_stats(51 pgs tid 402 v 599) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:59.785996 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 prepare_update pg_stats(51 pgs tid 402 v 599) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:33:59.785999 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 prepare_pg_stats pg_stats(51 pgs tid 402 v 599) v1 from osd.0 2013-04-03 10:33:59.786005 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 got osd.0 osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:33:59.827582 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 2013-04-03 10:33:59.827589 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 2013-04-03 10:33:59.827615 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 preprocess_query pg_stats(71 pgs tid 521 v 599) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:59.827622 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 prepare_update pg_stats(71 pgs tid 521 v 599) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:33:59.827625 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 prepare_pg_stats pg_stats(71 pgs tid 521 v 599) v1 from osd.2 2013-04-03 10:33:59.827629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:34:00.492382 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393969 encode_pending v393970 2013-04-03 10:34:00.518326 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28963 encode_pending v 28964 2013-04-03 10:34:00.620184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos 2013-04-03 10:34:00.620213 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos version 393970 summary v 393969 2013-04-03 10:34:00.620227 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos latest full 393798 2013-04-03 10:34:00.620245 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393970 update_from_paxos applying incremental log 393970 2013-04-03 10:33:56.135933 mon.0 10.252.24.25:6789/0 14 : [INF] pgmap v28962: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:00.620270 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393970 update_from_paxos applying incremental log 393970 2013-04-03 10:33:59.493020 mon.0 10.252.24.25:6789/0 15 : [INF] pgmap v28963: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:00.620309 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 check_subs 2013-04-03 10:34:00.620337 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 create_pending v 393971 2013-04-03 10:34:00.620358 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393970 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:00.734251 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:00.734259 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fcd80 from 10.252.24.88:6789/0 2013-04-03 10:34:00.922046 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28963 update_from_paxos applying incremental 28964 2013-04-03 10:34:00.922349 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 v28964: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5672 MB used, 1067 GB / 1079 GB avail; 3608B/s wr, 0op/s 2013-04-03 10:34:00.922384 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 send_pg_creates to 0 pgs 2013-04-03 10:34:00.922389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 update_logger 2013-04-03 10:34:00.922415 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 create_pending v 28965 2013-04-03 10:34:00.922427 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28964 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:00.922453 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28964 _updated_stats for osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:00.922523 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 preprocess_query pg_stats(45 pgs tid 2 v 599) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:00.922531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 prepare_update pg_stats(45 pgs tid 2 v 599) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:00.922535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 prepare_pg_stats pg_stats(45 pgs tid 2 v 599) v1 from osd.1 2013-04-03 10:34:00.922539 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 got osd.1 osd_stat(1113 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1110 MB used, 148 GB avail, 149 GB total, peers []/[])) 2013-04-03 10:34:00.922595 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 check_osd_map already seen 599 >= 599 2013-04-03 10:34:00.922600 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 update_logger 2013-04-03 10:34:00.922613 7fcdffcdb700 0 log [INF] : pgmap v28964: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5672 MB used, 1067 GB / 1079 GB avail; 3608B/s wr, 0op/s 2013-04-03 10:34:00.922669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos 2013-04-03 10:34:00.922686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos version 393970 summary v 393970 2013-04-03 10:34:00.922703 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:00.922744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos 2013-04-03 10:34:00.922759 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos version 393970 summary v 393970 2013-04-03 10:34:00.922767 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:00.922779 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:00.922797 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:00.922809 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:00.922820 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 logging 2013-04-03 10:34:00.922614 mon.0 10.252.24.25:6789/0 16 : [INF] pgmap v28964: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5672 MB used, 1067 GB / 1079 GB avail; 3608B/s wr, 0op/s 2013-04-03 10:34:00.954391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos 2013-04-03 10:34:00.954409 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos version 393970 summary v 393970 2013-04-03 10:34:00.954423 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:00.957796 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:00.957805 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc6c0 from 10.252.24.88:6789/0 2013-04-03 10:34:00.957852 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 preprocess_query pg_stats(73 pgs tid 30430 v 599) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:00.957860 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 prepare_update pg_stats(73 pgs tid 30430 v 599) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:00.957863 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 prepare_pg_stats pg_stats(73 pgs tid 30430 v 599) v1 from osd.3 2013-04-03 10:34:00.957868 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 got osd.3 osd_stat(1190 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1187 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:34:01.375848 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21664 v184) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:34:01.920604 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28964 encode_pending v 28965 2013-04-03 10:34:01.922961 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393970 encode_pending v393971 2013-04-03 10:34:01.955860 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28964 update_from_paxos applying incremental 28965 2013-04-03 10:34:01.956150 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 v28965: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 6631B/s wr, 0op/s 2013-04-03 10:34:01.956184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 send_pg_creates to 0 pgs 2013-04-03 10:34:01.956188 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 update_logger 2013-04-03 10:34:01.956211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 create_pending v 28966 2013-04-03 10:34:01.956222 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28965 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:01.956262 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28965 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:01.956297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 check_osd_map already seen 599 >= 599 2013-04-03 10:34:01.956301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 update_logger 2013-04-03 10:34:01.956312 7fcdffcdb700 0 log [INF] : pgmap v28965: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 6631B/s wr, 0op/s 2013-04-03 10:34:02.157463 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos 2013-04-03 10:34:02.157498 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393970 2013-04-03 10:34:02.157512 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos latest full 393798 2013-04-03 10:34:02.157529 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393971 update_from_paxos applying incremental log 393971 2013-04-03 10:34:00.922614 mon.0 10.252.24.25:6789/0 16 : [INF] pgmap v28964: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5672 MB used, 1067 GB / 1079 GB avail; 3608B/s wr, 0op/s 2013-04-03 10:34:02.157571 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 check_subs 2013-04-03 10:34:02.157601 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 create_pending v 393972 2013-04-03 10:34:02.157622 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393971 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:02.157654 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos 2013-04-03 10:34:02.157668 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393971 2013-04-03 10:34:02.157677 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:02.157689 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:02.157707 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 nothing new 2013-04-03 10:34:02.157755 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos 2013-04-03 10:34:02.157770 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393971 2013-04-03 10:34:02.157787 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:02.232389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos 2013-04-03 10:34:02.232405 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393971 2013-04-03 10:34:02.232419 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:02.754726 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:02.754736 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e48c0 from 10.252.24.88:6789/0 2013-04-03 10:34:02.754785 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 6 v184) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:03.321182 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28965 check_down_pgs 2013-04-03 10:34:03.321214 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28965 v28965: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 6631B/s wr, 0op/s 2013-04-03 10:34:03.321239 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e184 e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby 2013-04-03 10:34:03.321260 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in 2013-04-03 10:34:03.321298 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 2013-04-03 10:34:03.321335 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos 2013-04-03 10:34:03.321353 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393971 2013-04-03 10:34:03.321362 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393971 log 2013-04-03 10:34:03.321378 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:03.321395 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:34:03.575627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140294 v184) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:34:04.823363 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:04.823372 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc6c0 from 10.252.24.88:6789/0 2013-04-03 10:34:04.823421 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 preprocess_query pg_stats(1 pgs tid 522 v 599) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:04.823429 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 prepare_update pg_stats(1 pgs tid 522 v 599) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:04.823433 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 prepare_pg_stats pg_stats(1 pgs tid 522 v 599) v1 from osd.2 2013-04-03 10:34:04.823438 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[])) 2013-04-03 10:34:04.873625 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28965 encode_pending v 28966 2013-04-03 10:34:04.902090 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28965 update_from_paxos applying incremental 28966 2013-04-03 10:34:04.902106 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 v28966: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 940B/s wr, 0op/s 2013-04-03 10:34:04.902124 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 send_pg_creates to 0 pgs 2013-04-03 10:34:04.902128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 update_logger 2013-04-03 10:34:04.902138 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 create_pending v 28967 2013-04-03 10:34:04.902146 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28966 _updated_stats for osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:04.902162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 check_osd_map already seen 599 >= 599 2013-04-03 10:34:04.902164 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 update_logger 2013-04-03 10:34:04.902176 7fcdffcdb700 0 log [INF] : pgmap v28966: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 940B/s wr, 0op/s 2013-04-03 10:34:04.902241 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos 2013-04-03 10:34:04.902258 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393971 2013-04-03 10:34:04.902276 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:04.902319 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos 2013-04-03 10:34:04.902334 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393971 2013-04-03 10:34:04.902343 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:04.902353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:34:04.902380 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:04.902391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:34:04.902401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 logging 2013-04-03 10:34:01.956313 mon.0 10.252.24.25:6789/0 17 : [INF] pgmap v28965: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 6631B/s wr, 0op/s 2013-04-03 10:34:04.902424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 logging 2013-04-03 10:34:04.902177 mon.0 10.252.24.25:6789/0 18 : [INF] pgmap v28966: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 940B/s wr, 0op/s 2013-04-03 10:34:04.923298 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos 2013-04-03 10:34:04.923314 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393971 2013-04-03 10:34:04.923329 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:05.375972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21665 v184) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:34:05.901241 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393971 encode_pending v393972 2013-04-03 10:34:06.072384 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos 2013-04-03 10:34:06.072418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393971 2013-04-03 10:34:06.072434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos latest full 393798 2013-04-03 10:34:06.072454 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393972 update_from_paxos applying incremental log 393972 2013-04-03 10:34:01.956313 mon.0 10.252.24.25:6789/0 17 : [INF] pgmap v28965: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 6631B/s wr, 0op/s 2013-04-03 10:34:06.072479 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393972 update_from_paxos applying incremental log 393972 2013-04-03 10:34:04.902177 mon.0 10.252.24.25:6789/0 18 : [INF] pgmap v28966: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 940B/s wr, 0op/s 2013-04-03 10:34:06.072514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 check_subs 2013-04-03 10:34:06.072543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 create_pending v 393973 2013-04-03 10:34:06.072565 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393972 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:06.072625 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos 2013-04-03 10:34:06.072640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393972 2013-04-03 10:34:06.072657 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:06.097612 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos 2013-04-03 10:34:06.097628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393972 2013-04-03 10:34:06.097642 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:06.754979 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:06.754990 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4340 from 10.252.24.88:6789/0 2013-04-03 10:34:06.755039 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 7 v184) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:07.575767 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140295 v184) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:34:08.321534 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28966 check_down_pgs 2013-04-03 10:34:08.321572 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28966 v28966: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 940B/s wr, 0op/s 2013-04-03 10:34:08.321598 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e184 e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby 2013-04-03 10:34:08.321619 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in 2013-04-03 10:34:08.321653 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 2013-04-03 10:34:08.321686 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos 2013-04-03 10:34:08.321704 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393972 2013-04-03 10:34:08.321713 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393972 log 2013-04-03 10:34:08.321729 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:08.321747 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:34:09.097126 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos 2013-04-03 10:34:09.097145 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393972 2013-04-03 10:34:09.097161 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:09.394536 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21666 v184) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:34:09.807027 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:09.807037 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad240 from 10.252.24.88:6789/0 2013-04-03 10:34:09.807088 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 preprocess_query pg_stats(4 pgs tid 403 v 599) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:09.807095 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 prepare_update pg_stats(4 pgs tid 403 v 599) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:09.807100 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 prepare_pg_stats pg_stats(4 pgs tid 403 v 599) v1 from osd.0 2013-04-03 10:34:09.807106 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 got osd.0 osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:34:09.847091 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:09.847100 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 2013-04-03 10:34:09.847149 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 preprocess_query pg_stats(3 pgs tid 523 v 599) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:09.847156 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 prepare_update pg_stats(3 pgs tid 523 v 599) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:09.847160 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 prepare_pg_stats pg_stats(3 pgs tid 523 v 599) v1 from osd.2 2013-04-03 10:34:09.847165 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[])) 2013-04-03 10:34:09.857252 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28966 encode_pending v 28967 2013-04-03 10:34:09.939299 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28966 update_from_paxos applying incremental 28967 2013-04-03 10:34:09.939338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 v28967: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 516B/s wr, 0op/s 2013-04-03 10:34:09.939358 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 send_pg_creates to 0 pgs 2013-04-03 10:34:09.939363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 update_logger 2013-04-03 10:34:09.939376 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 create_pending v 28968 2013-04-03 10:34:09.939388 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28967 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:09.939407 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28967 _updated_stats for osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:09.939417 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 check_osd_map already seen 599 >= 599 2013-04-03 10:34:09.939421 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 update_logger 2013-04-03 10:34:09.939431 7fcdffcdb700 0 log [INF] : pgmap v28967: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 516B/s wr, 0op/s 2013-04-03 10:34:09.939488 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos 2013-04-03 10:34:09.939507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393972 2013-04-03 10:34:09.939523 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:09.939563 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos 2013-04-03 10:34:09.939578 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393972 2013-04-03 10:34:09.939585 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:09.939597 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:09.939616 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:09.939627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:09.939637 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 logging 2013-04-03 10:34:09.939432 mon.0 10.252.24.25:6789/0 19 : [INF] pgmap v28967: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 516B/s wr, 0op/s 2013-04-03 10:34:09.993744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos 2013-04-03 10:34:09.993764 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393972 2013-04-03 10:34:09.993781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:10.582686 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:10.582696 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad240 from 10.252.24.88:6789/0 2013-04-03 10:34:10.582746 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 preprocess_query pg_stats(5 pgs tid 3 v 599) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:10.582753 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 prepare_update pg_stats(5 pgs tid 3 v 599) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:10.582757 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 prepare_pg_stats pg_stats(5 pgs tid 3 v 599) v1 from osd.1 2013-04-03 10:34:10.582763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 got osd.1 osd_stat(1113 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1113 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:34:10.733762 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:10.733771 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fcd80 from 10.252.24.88:6789/0 2013-04-03 10:34:10.733819 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 preprocess_query pg_stats(4 pgs tid 30431 v 599) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:10.733827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 prepare_update pg_stats(4 pgs tid 30431 v 599) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:10.733831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 prepare_pg_stats pg_stats(4 pgs tid 30431 v 599) v1 from osd.3 2013-04-03 10:34:10.733836 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 got osd.3 osd_stat(1191 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1190 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:34:10.767831 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:10.767838 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4340 from 10.252.24.88:6789/0 2013-04-03 10:34:10.767871 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 8 v184) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:10.938999 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393972 encode_pending v393973 2013-04-03 10:34:10.940159 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28967 encode_pending v 28968 2013-04-03 10:34:11.121798 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos 2013-04-03 10:34:11.121834 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos version 393973 summary v 393972 2013-04-03 10:34:11.121848 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos latest full 393798 2013-04-03 10:34:11.121865 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393973 update_from_paxos applying incremental log 393973 2013-04-03 10:34:09.939432 mon.0 10.252.24.25:6789/0 19 : [INF] pgmap v28967: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 516B/s wr, 0op/s 2013-04-03 10:34:11.121907 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 check_subs 2013-04-03 10:34:11.121936 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 create_pending v 393974 2013-04-03 10:34:11.121957 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393973 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:11.201962 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28967 update_from_paxos applying incremental 28968 2013-04-03 10:34:11.202003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 v28968: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:11.202023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 send_pg_creates to 0 pgs 2013-04-03 10:34:11.202027 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 update_logger 2013-04-03 10:34:11.202039 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 create_pending v 28969 2013-04-03 10:34:11.202047 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28968 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:11.202065 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28968 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:11.202104 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 check_osd_map already seen 599 >= 599 2013-04-03 10:34:11.202108 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 update_logger 2013-04-03 10:34:11.202119 7fcdffcdb700 0 log [INF] : pgmap v28968: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:11.202176 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos 2013-04-03 10:34:11.202193 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos version 393973 summary v 393973 2013-04-03 10:34:11.202213 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:11.202254 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos 2013-04-03 10:34:11.202268 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos version 393973 summary v 393973 2013-04-03 10:34:11.202277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:11.202288 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:11.202308 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:11.202321 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:11.202331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 logging 2013-04-03 10:34:11.202120 mon.0 10.252.24.25:6789/0 20 : [INF] pgmap v28968: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:11.252151 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos 2013-04-03 10:34:11.252167 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos version 393973 summary v 393973 2013-04-03 10:34:11.252181 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:11.586508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140296 v184) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:34:12.201180 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393973 encode_pending v393974 2013-04-03 10:34:12.369635 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos 2013-04-03 10:34:12.369662 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393973 2013-04-03 10:34:12.369675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos latest full 393798 2013-04-03 10:34:12.369693 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393974 update_from_paxos applying incremental log 393974 2013-04-03 10:34:11.202120 mon.0 10.252.24.25:6789/0 20 : [INF] pgmap v28968: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:12.369736 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 check_subs 2013-04-03 10:34:12.369765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 create_pending v 393975 2013-04-03 10:34:12.369786 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393974 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:12.369847 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos 2013-04-03 10:34:12.369862 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393974 2013-04-03 10:34:12.369879 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:12.406317 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos 2013-04-03 10:34:12.406333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393974 2013-04-03 10:34:12.406347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:13.321873 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28968 check_down_pgs 2013-04-03 10:34:13.321897 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28968 v28968: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:13.321922 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e184 e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby 2013-04-03 10:34:13.321947 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in 2013-04-03 10:34:13.321984 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 2013-04-03 10:34:13.322013 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos 2013-04-03 10:34:13.322030 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393974 2013-04-03 10:34:13.322039 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393974 log 2013-04-03 10:34:13.322058 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:13.322076 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:34:13.386220 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21667 v184) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:34:14.780202 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:14.780212 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e48c0 from 10.252.24.88:6789/0 2013-04-03 10:34:14.780261 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 9 v184) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:14.810254 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:14.810260 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fcd80 from 10.252.24.88:6789/0 2013-04-03 10:34:14.810292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 preprocess_query pg_stats(3 pgs tid 404 v 599) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:14.810299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 prepare_update pg_stats(3 pgs tid 404 v 599) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:14.810303 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 prepare_pg_stats pg_stats(3 pgs tid 404 v 599) v1 from osd.0 2013-04-03 10:34:14.810309 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 got osd.0 osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:34:14.845275 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:14.845282 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad900 from 10.252.24.88:6789/0 2013-04-03 10:34:14.845319 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 preprocess_query pg_stats(7 pgs tid 524 v 599) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:14.845327 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 prepare_update pg_stats(7 pgs tid 524 v 599) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:14.845330 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 prepare_pg_stats pg_stats(7 pgs tid 524 v 599) v1 from osd.2 2013-04-03 10:34:14.845336 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[])) 2013-04-03 10:34:14.860473 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28968 encode_pending v 28969 2013-04-03 10:34:15.027294 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28968 update_from_paxos applying incremental 28969 2013-04-03 10:34:15.027338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 v28969: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2456B/s wr, 0op/s 2013-04-03 10:34:15.027359 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 send_pg_creates to 0 pgs 2013-04-03 10:34:15.027363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 update_logger 2013-04-03 10:34:15.027375 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 create_pending v 28970 2013-04-03 10:34:15.027384 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28969 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:15.027401 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28969 _updated_stats for osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:15.027435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 check_osd_map already seen 599 >= 599 2013-04-03 10:34:15.027439 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 update_logger 2013-04-03 10:34:15.027450 7fcdffcdb700 0 log [INF] : pgmap v28969: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2456B/s wr, 0op/s 2013-04-03 10:34:15.027519 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos 2013-04-03 10:34:15.027536 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393974 2013-04-03 10:34:15.027554 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:15.027597 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos 2013-04-03 10:34:15.027611 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393974 2013-04-03 10:34:15.027620 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:15.027632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:15.027653 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:15.027665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:15.027675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:15.027457 mon.0 10.252.24.25:6789/0 21 : [INF] pgmap v28969: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2456B/s wr, 0op/s 2013-04-03 10:34:15.243333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos 2013-04-03 10:34:15.243350 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393974 2013-04-03 10:34:15.243364 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:15.568487 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:15.568497 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fcd80 from 10.252.24.88:6789/0 2013-04-03 10:34:15.568549 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 preprocess_query pg_stats(1 pgs tid 4 v 599) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:15.568556 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 prepare_update pg_stats(1 pgs tid 4 v 599) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:15.568560 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 prepare_pg_stats pg_stats(1 pgs tid 4 v 599) v1 from osd.1 2013-04-03 10:34:15.568567 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 got osd.1 osd_stat(1113 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1113 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:34:15.591971 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140297 v184) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:34:15.738075 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:15.738084 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad240 from 10.252.24.88:6789/0 2013-04-03 10:34:15.738132 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 preprocess_query pg_stats(2 pgs tid 30432 v 599) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:15.738140 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 prepare_update pg_stats(2 pgs tid 30432 v 599) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:15.738144 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 prepare_pg_stats pg_stats(2 pgs tid 30432 v 599) v1 from osd.3 2013-04-03 10:34:15.738150 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 got osd.3 osd_stat(1191 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1191 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:34:15.784627 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:15.784636 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24e9200 from 10.252.24.88:6789/0 2013-04-03 10:34:15.784660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos 2013-04-03 10:34:15.784675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393974 2013-04-03 10:34:15.784686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 preprocess_query log(23 entries) v1 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:15.784697 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 preprocess_log log(23 entries) v1 from mds.0 2013-04-03 10:34:15.784726 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 prepare_update log(23 entries) v1 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:15.784738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 prepare_log log(23 entries) v1 from mds.0 2013-04-03 10:34:15.784748 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.962981 mds.0 10.252.24.25:6802/3144 1 : [WRN] replayed op client.10161:60,59 used ino 10000008c2b but session next is 1000000841c 2013-04-03 10:34:15.784763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.963022 mds.0 10.252.24.25:6802/3144 2 : [WRN] replayed op client.10161:62,59 used ino 10000008c2c but session next is 1000000841c 2013-04-03 10:34:15.784777 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.963132 mds.0 10.252.24.25:6802/3144 3 : [WRN] replayed op client.10161:66,59 used ino 10000008c2d but session next is 1000000841c 2013-04-03 10:34:15.784789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.963171 mds.0 10.252.24.25:6802/3144 4 : [WRN] replayed op client.10161:67,59 used ino 10000008c2e but session next is 1000000841c 2013-04-03 10:34:15.784804 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.963212 mds.0 10.252.24.25:6802/3144 5 : [WRN] replayed op client.10161:69,59 used ino 10000008c2f but session next is 1000000841c 2013-04-03 10:34:15.784819 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966144 mds.0 10.252.24.25:6802/3144 6 : [WRN] replayed op client.10004:112,111 used ino 10000008c3f but session next is 1000000883d 2013-04-03 10:34:15.784834 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966184 mds.0 10.252.24.25:6802/3144 7 : [WRN] replayed op client.10004:114,111 used ino 10000008c40 but session next is 1000000883d 2013-04-03 10:34:15.784849 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966293 mds.0 10.252.24.25:6802/3144 8 : [WRN] replayed op client.10004:118,111 used ino 10000008c41 but session next is 1000000883d 2013-04-03 10:34:15.784865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966333 mds.0 10.252.24.25:6802/3144 9 : [WRN] replayed op client.10004:119,111 used ino 10000008c42 but session next is 1000000883d 2013-04-03 10:34:15.784880 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966373 mds.0 10.252.24.25:6802/3144 10 : [WRN] replayed op client.10004:121,111 used ino 10000008c43 but session next is 1000000883d 2013-04-03 10:34:15.784897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966759 mds.0 10.252.24.25:6802/3144 11 : [WRN] replayed op client.10004:137,136 used ino 1000000883e but session next is 10000008428 2013-04-03 10:34:15.784913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966796 mds.0 10.252.24.25:6802/3144 12 : [WRN] replayed op client.10004:139,136 used ino 1000000883f but session next is 10000008428 2013-04-03 10:34:15.784929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966903 mds.0 10.252.24.25:6802/3144 13 : [WRN] replayed op client.10004:143,136 used ino 10000008840 but session next is 10000008428 2013-04-03 10:34:15.784945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966943 mds.0 10.252.24.25:6802/3144 14 : [WRN] replayed op client.10004:144,136 used ino 10000008841 but session next is 10000008428 2013-04-03 10:34:15.784962 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966983 mds.0 10.252.24.25:6802/3144 15 : [WRN] replayed op client.10004:146,136 used ino 10000008842 but session next is 10000008428 2013-04-03 10:34:15.784979 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.968711 mds.0 10.252.24.25:6802/3144 16 : [WRN] replayed op client.10004:229,217 used ino 10000008c44 but session next is 10000008437 2013-04-03 10:34:15.784995 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.968753 mds.0 10.252.24.25:6802/3144 17 : [WRN] replayed op client.10004:231,217 used ino 10000008c45 but session next is 10000008437 2013-04-03 10:34:15.785012 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.973451 mds.0 10.252.24.25:6802/3144 18 : [WRN] replayed op client.10004:379,356 used ino 10000008c46 but session next is 1000000844c 2013-04-03 10:34:15.785029 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.973500 mds.0 10.252.24.25:6802/3144 19 : [WRN] replayed op client.10004:380,356 used ino 10000008c47 but session next is 1000000844c 2013-04-03 10:34:15.785045 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.973547 mds.0 10.252.24.25:6802/3144 20 : [WRN] replayed op client.10004:381,356 used ino 10000008c48 but session next is 1000000844c 2013-04-03 10:34:15.785065 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.973592 mds.0 10.252.24.25:6802/3144 21 : [WRN] replayed op client.10004:382,356 used ino 10000008c49 but session next is 1000000844c 2013-04-03 10:34:15.785084 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.994218 mds.0 10.252.24.25:6802/3144 22 : [WRN] replayed op client.10448:72,70 used ino 10000008ceb but session next is 100000088f7 2013-04-03 10:34:15.785101 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.994380 mds.0 10.252.24.25:6802/3144 23 : [WRN] replayed op client.10448:88,86 used ino 100000088f8 but session next is 100000084c2 2013-04-03 10:34:16.025863 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393974 encode_pending v393975 2013-04-03 10:34:16.027386 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28969 encode_pending v 28970 2013-04-03 10:34:16.140014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.140041 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393974 2013-04-03 10:34:16.140054 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos latest full 393798 2013-04-03 10:34:16.140073 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.962981 mds.0 10.252.24.25:6802/3144 1 : [WRN] replayed op client.10161:60,59 used ino 10000008c2b but session next is 1000000841c 2013-04-03 10:34:16.140097 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.963022 mds.0 10.252.24.25:6802/3144 2 : [WRN] replayed op client.10161:62,59 used ino 10000008c2c but session next is 1000000841c 2013-04-03 10:34:16.140116 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.963132 mds.0 10.252.24.25:6802/3144 3 : [WRN] replayed op client.10161:66,59 used ino 10000008c2d but session next is 1000000841c 2013-04-03 10:34:16.140134 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.963171 mds.0 10.252.24.25:6802/3144 4 : [WRN] replayed op client.10161:67,59 used ino 10000008c2e but session next is 1000000841c 2013-04-03 10:34:16.140153 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.963212 mds.0 10.252.24.25:6802/3144 5 : [WRN] replayed op client.10161:69,59 used ino 10000008c2f but session next is 1000000841c 2013-04-03 10:34:16.140174 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966144 mds.0 10.252.24.25:6802/3144 6 : [WRN] replayed op client.10004:112,111 used ino 10000008c3f but session next is 1000000883d 2013-04-03 10:34:16.140197 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966184 mds.0 10.252.24.25:6802/3144 7 : [WRN] replayed op client.10004:114,111 used ino 10000008c40 but session next is 1000000883d 2013-04-03 10:34:16.140216 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966293 mds.0 10.252.24.25:6802/3144 8 : [WRN] replayed op client.10004:118,111 used ino 10000008c41 but session next is 1000000883d 2013-04-03 10:34:16.140234 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966333 mds.0 10.252.24.25:6802/3144 9 : [WRN] replayed op client.10004:119,111 used ino 10000008c42 but session next is 1000000883d 2013-04-03 10:34:16.140251 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966373 mds.0 10.252.24.25:6802/3144 10 : [WRN] replayed op client.10004:121,111 used ino 10000008c43 but session next is 1000000883d 2013-04-03 10:34:16.140270 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966759 mds.0 10.252.24.25:6802/3144 11 : [WRN] replayed op client.10004:137,136 used ino 1000000883e but session next is 10000008428 2013-04-03 10:34:16.140288 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966796 mds.0 10.252.24.25:6802/3144 12 : [WRN] replayed op client.10004:139,136 used ino 1000000883f but session next is 10000008428 2013-04-03 10:34:16.140305 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966903 mds.0 10.252.24.25:6802/3144 13 : [WRN] replayed op client.10004:143,136 used ino 10000008840 but session next is 10000008428 2013-04-03 10:34:16.140323 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966943 mds.0 10.252.24.25:6802/3144 14 : [WRN] replayed op client.10004:144,136 used ino 10000008841 but session next is 10000008428 2013-04-03 10:34:16.140341 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966983 mds.0 10.252.24.25:6802/3144 15 : [WRN] replayed op client.10004:146,136 used ino 10000008842 but session next is 10000008428 2013-04-03 10:34:16.140358 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.968711 mds.0 10.252.24.25:6802/3144 16 : [WRN] replayed op client.10004:229,217 used ino 10000008c44 but session next is 10000008437 2013-04-03 10:34:16.140376 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.968753 mds.0 10.252.24.25:6802/3144 17 : [WRN] replayed op client.10004:231,217 used ino 10000008c45 but session next is 10000008437 2013-04-03 10:34:16.140394 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.973451 mds.0 10.252.24.25:6802/3144 18 : [WRN] replayed op client.10004:379,356 used ino 10000008c46 but session next is 1000000844c 2013-04-03 10:34:16.140411 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.973500 mds.0 10.252.24.25:6802/3144 19 : [WRN] replayed op client.10004:380,356 used ino 10000008c47 but session next is 1000000844c 2013-04-03 10:34:16.140429 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.973547 mds.0 10.252.24.25:6802/3144 20 : [WRN] replayed op client.10004:381,356 used ino 10000008c48 but session next is 1000000844c 2013-04-03 10:34:16.140446 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.973592 mds.0 10.252.24.25:6802/3144 21 : [WRN] replayed op client.10004:382,356 used ino 10000008c49 but session next is 1000000844c 2013-04-03 10:34:16.140464 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.994218 mds.0 10.252.24.25:6802/3144 22 : [WRN] replayed op client.10448:72,70 used ino 10000008ceb but session next is 100000088f7 2013-04-03 10:34:16.140482 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.994380 mds.0 10.252.24.25:6802/3144 23 : [WRN] replayed op client.10448:88,86 used ino 100000088f8 but session next is 100000084c2 2013-04-03 10:34:16.140503 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:15.027457 mon.0 10.252.24.25:6789/0 21 : [INF] pgmap v28969: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2456B/s wr, 0op/s 2013-04-03 10:34:16.140550 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 check_subs 2013-04-03 10:34:16.140583 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 create_pending v 393976 2013-04-03 10:34:16.140605 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:16.140625 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 _updated_log for mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:16.498366 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28969 update_from_paxos applying incremental 28970 2013-04-03 10:34:16.498392 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s 2013-04-03 10:34:16.498413 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 send_pg_creates to 0 pgs 2013-04-03 10:34:16.498416 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 update_logger 2013-04-03 10:34:16.498428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 create_pending v 28971 2013-04-03 10:34:16.498439 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28970 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:16.498457 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28970 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:16.498495 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 check_osd_map already seen 599 >= 599 2013-04-03 10:34:16.498499 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 update_logger 2013-04-03 10:34:16.498511 7fcdffcdb700 0 log [INF] : pgmap v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s 2013-04-03 10:34:16.498570 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.498588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 2013-04-03 10:34:16.498605 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:16.498646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.498660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 2013-04-03 10:34:16.498669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:16.498685 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:16.498702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:16.498713 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:16.498723 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.498512 mon.0 10.252.24.25:6789/0 22 : [INF] pgmap v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s 2013-04-03 10:34:16.523927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.523943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 2013-04-03 10:34:16.523957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:16.718593 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:16.718603 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e3000 from 10.252.24.88:6789/0 2013-04-03 10:34:16.718651 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:reconnect seq 10 v184) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:16.718666 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e184 prepare_update mdsbeacon(11234/hotcpc9039 up:reconnect seq 10 v184) v2 2013-04-03 10:34:16.718672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 prepare_beacon mds.0 up:replay -> up:reconnect standby_for_rank=-1 2013-04-03 10:34:16.718674 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e184 prepare_beacon pending map now: 2013-04-03 10:34:16.718677 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e184 print_map epoch 185 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:33:42.867413 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 596 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:reconnect seq 10 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11006: 10.252.24.88:6800/17213 'hotcpc12533' mds.-1.0 up:standby seq 1 2013-04-03 10:34:16.718711 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 encode_pending e185 2013-04-03 10:34:16.820742 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 update_from_paxos version 185, my e 184 2013-04-03 10:34:16.820769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 update_from_paxos got 185 2013-04-03 10:34:16.820784 7fcdffcdb700 4 mon.hotcpc9039@0(leader).mds e185 new map 2013-04-03 10:34:16.820787 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e185 print_map epoch 185 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:34:16.718715 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 596 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:reconnect seq 10 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11006: 10.252.24.88:6800/17213 'hotcpc12533' mds.-1.0 up:standby seq 1 2013-04-03 10:34:16.820837 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 update_logger 2013-04-03 10:34:16.820859 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 create_pending e186 2013-04-03 10:34:16.820872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 _updated mds.0 mdsbeacon(11234/hotcpc9039 up:reconnect seq 10 v184) v2 2013-04-03 10:34:16.820882 7fcdffcdb700 0 log [INF] : mds.0 10.252.24.25:6802/3144 up:reconnect 2013-04-03 10:34:16.820910 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 e185: 1/1/1 up {0=hotcpc9039=up:reconnect}, 2 up:standby 2013-04-03 10:34:16.820923 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 update_logger 2013-04-03 10:34:16.820928 7fcdffcdb700 0 log [INF] : mdsmap e185: 1/1/1 up {0=hotcpc9039=up:reconnect}, 2 up:standby 2013-04-03 10:34:16.820992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.821010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 2013-04-03 10:34:16.821027 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:16.821068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.821082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 2013-04-03 10:34:16.821091 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:16.821103 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:16.821120 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 nothing new 2013-04-03 10:34:16.821147 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.821161 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 2013-04-03 10:34:16.821173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:16.821183 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:34:16.821202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:16.821214 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:34:16.821223 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.498512 mon.0 10.252.24.25:6789/0 22 : [INF] pgmap v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s 2013-04-03 10:34:16.821239 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.820883 mon.0 10.252.24.25:6789/0 23 : [INF] mds.0 10.252.24.25:6802/3144 up:reconnect 2013-04-03 10:34:16.832672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.832689 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 2013-04-03 10:34:16.832704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:16.860349 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:16.860356 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2146600 from 10.252.24.88:6789/0 2013-04-03 10:34:16.860390 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 preprocess_query mdsbeacon(11234/hotcpc9039 up:rejoin seq 11 v185) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:16.860399 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e185 prepare_update mdsbeacon(11234/hotcpc9039 up:rejoin seq 11 v185) v2 2013-04-03 10:34:16.860402 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 prepare_beacon mds.0 up:reconnect -> up:rejoin standby_for_rank=-1 2013-04-03 10:34:16.860404 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e185 prepare_beacon pending map now: 2013-04-03 10:34:16.860405 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e185 print_map epoch 186 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:34:16.718715 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 596 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:rejoin seq 11 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11006: 10.252.24.88:6800/17213 'hotcpc12533' mds.-1.0 up:standby seq 1 2013-04-03 10:34:16.860434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 encode_pending e186 2013-04-03 10:34:16.888343 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 update_from_paxos version 186, my e 185 2013-04-03 10:34:16.888357 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 update_from_paxos got 186 2013-04-03 10:34:16.888367 7fcdffcdb700 4 mon.hotcpc9039@0(leader).mds e186 new map 2013-04-03 10:34:16.888368 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e186 print_map epoch 186 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:34:16.860438 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 596 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:rejoin seq 11 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11006: 10.252.24.88:6800/17213 'hotcpc12533' mds.-1.0 up:standby seq 1 2013-04-03 10:34:16.888414 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 update_logger 2013-04-03 10:34:16.888428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 create_pending e187 2013-04-03 10:34:16.888438 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 _updated mds.0 mdsbeacon(11234/hotcpc9039 up:rejoin seq 11 v185) v2 2013-04-03 10:34:16.888445 7fcdffcdb700 0 log [INF] : mds.0 10.252.24.25:6802/3144 up:rejoin 2013-04-03 10:34:16.888475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 e186: 1/1/1 up {0=hotcpc9039=up:rejoin}, 2 up:standby 2013-04-03 10:34:16.888484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 update_logger 2013-04-03 10:34:16.888490 7fcdffcdb700 0 log [INF] : mdsmap e186: 1/1/1 up {0=hotcpc9039=up:rejoin}, 2 up:standby 2013-04-03 10:34:16.888543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.888559 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 2013-04-03 10:34:16.888575 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:16.888613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.888628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 2013-04-03 10:34:16.888636 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:16.888646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:34:16.888666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:16.888682 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:34:16.888691 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.820883 mon.0 10.252.24.25:6789/0 23 : [INF] mds.0 10.252.24.25:6802/3144 up:reconnect 2013-04-03 10:34:16.888708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.820929 mon.0 10.252.24.25:6789/0 24 : [INF] mdsmap e185: 1/1/1 up {0=hotcpc9039=up:reconnect}, 2 up:standby 2013-04-03 10:34:16.888754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.888768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 2013-04-03 10:34:16.888777 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:16.888787 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:34:16.888805 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:16.888817 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:34:16.888826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.820929 mon.0 10.252.24.25:6789/0 24 : [INF] mdsmap e185: 1/1/1 up {0=hotcpc9039=up:reconnect}, 2 up:standby 2013-04-03 10:34:16.888842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.888446 mon.0 10.252.24.25:6789/0 25 : [INF] mds.0 10.252.24.25:6802/3144 up:rejoin 2013-04-03 10:34:16.914465 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.914481 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 2013-04-03 10:34:16.914495 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:16.921722 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:16.921731 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2146080 from 10.252.24.88:6789/0 2013-04-03 10:34:16.921780 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 12 v186) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:16.921790 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e186 prepare_update mdsbeacon(11234/hotcpc9039 up:active seq 12 v186) v2 2013-04-03 10:34:16.921793 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 prepare_beacon mds.0 up:rejoin -> up:active standby_for_rank=-1 2013-04-03 10:34:16.921795 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e186 prepare_beacon pending map now: 2013-04-03 10:34:16.921796 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e186 print_map epoch 187 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:34:16.860438 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 596 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:active seq 12 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11006: 10.252.24.88:6800/17213 'hotcpc12533' mds.-1.0 up:standby seq 1 2013-04-03 10:34:16.921825 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 encode_pending e187 2013-04-03 10:34:16.935371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 update_from_paxos version 187, my e 186 2013-04-03 10:34:16.935386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 update_from_paxos got 187 2013-04-03 10:34:16.935397 7fcdffcdb700 4 mon.hotcpc9039@0(leader).mds e187 new map 2013-04-03 10:34:16.935400 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e187 print_map epoch 187 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:34:16.921828 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 596 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:active seq 12 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11006: 10.252.24.88:6800/17213 'hotcpc12533' mds.-1.0 up:standby seq 1 2013-04-03 10:34:16.935449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 update_logger 2013-04-03 10:34:16.935472 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 create_pending e188 2013-04-03 10:34:16.935484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 _updated mds.0 mdsbeacon(11234/hotcpc9039 up:active seq 12 v186) v2 2013-04-03 10:34:16.935494 7fcdffcdb700 0 log [INF] : mds.0 10.252.24.25:6802/3144 up:active 2013-04-03 10:34:16.935527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:34:16.935537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 update_logger 2013-04-03 10:34:16.935543 7fcdffcdb700 0 log [INF] : mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:34:16.935596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.935613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 2013-04-03 10:34:16.935631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:16.935673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.935687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 2013-04-03 10:34:16.935696 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:16.935706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:34:16.935726 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:16.935737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:34:16.935747 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.888446 mon.0 10.252.24.25:6789/0 25 : [INF] mds.0 10.252.24.25:6802/3144 up:rejoin 2013-04-03 10:34:16.935763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.888491 mon.0 10.252.24.25:6789/0 26 : [INF] mdsmap e186: 1/1/1 up {0=hotcpc9039=up:rejoin}, 2 up:standby 2013-04-03 10:34:16.935810 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.935824 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 2013-04-03 10:34:16.935832 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:16.935842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:34:16.935860 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:16.935872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:34:16.935881 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.888491 mon.0 10.252.24.25:6789/0 26 : [INF] mdsmap e186: 1/1/1 up {0=hotcpc9039=up:rejoin}, 2 up:standby 2013-04-03 10:34:16.935897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.935495 mon.0 10.252.24.25:6789/0 27 : [INF] mds.0 10.252.24.25:6802/3144 up:active 2013-04-03 10:34:16.961137 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos 2013-04-03 10:34:16.961153 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 2013-04-03 10:34:16.961167 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:17.376325 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21668 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:34:17.497930 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393975 encode_pending v393976 2013-04-03 10:34:17.537423 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos 2013-04-03 10:34:17.537457 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos version 393976 summary v 393975 2013-04-03 10:34:17.537471 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos latest full 393798 2013-04-03 10:34:17.537489 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 update_from_paxos applying incremental log 393976 2013-04-03 10:34:16.498512 mon.0 10.252.24.25:6789/0 22 : [INF] pgmap v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s 2013-04-03 10:34:17.537515 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 update_from_paxos applying incremental log 393976 2013-04-03 10:34:16.820883 mon.0 10.252.24.25:6789/0 23 : [INF] mds.0 10.252.24.25:6802/3144 up:reconnect 2013-04-03 10:34:17.537536 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 update_from_paxos applying incremental log 393976 2013-04-03 10:34:16.820929 mon.0 10.252.24.25:6789/0 24 : [INF] mdsmap e185: 1/1/1 up {0=hotcpc9039=up:reconnect}, 2 up:standby 2013-04-03 10:34:17.537555 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 update_from_paxos applying incremental log 393976 2013-04-03 10:34:16.888446 mon.0 10.252.24.25:6789/0 25 : [INF] mds.0 10.252.24.25:6802/3144 up:rejoin 2013-04-03 10:34:17.537573 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 update_from_paxos applying incremental log 393976 2013-04-03 10:34:16.888491 mon.0 10.252.24.25:6789/0 26 : [INF] mdsmap e186: 1/1/1 up {0=hotcpc9039=up:rejoin}, 2 up:standby 2013-04-03 10:34:17.537592 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 update_from_paxos applying incremental log 393976 2013-04-03 10:34:16.935495 mon.0 10.252.24.25:6789/0 27 : [INF] mds.0 10.252.24.25:6802/3144 up:active 2013-04-03 10:34:17.537638 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 check_subs 2013-04-03 10:34:17.537670 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 create_pending v 393977 2013-04-03 10:34:17.537691 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:17.537710 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:17.537728 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:17.537744 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:17.537760 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:17.537776 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:17.537844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos 2013-04-03 10:34:17.537860 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos version 393976 summary v 393976 2013-04-03 10:34:17.537878 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:17.565165 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos 2013-04-03 10:34:17.565181 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos version 393976 summary v 393976 2013-04-03 10:34:17.565196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:18.322222 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28970 check_down_pgs 2013-04-03 10:34:18.322255 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28970 v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s 2013-04-03 10:34:18.322281 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:34:18.322303 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in 2013-04-03 10:34:18.322340 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 2013-04-03 10:34:18.322368 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos 2013-04-03 10:34:18.322385 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos version 393976 summary v 393976 2013-04-03 10:34:18.322394 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393976 log 2013-04-03 10:34:18.322410 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:18.322428 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:34:19.576177 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140298 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:34:19.785610 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:19.785619 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 2013-04-03 10:34:19.785667 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 preprocess_query pg_stats(4 pgs tid 405 v 599) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:19.785675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 prepare_update pg_stats(4 pgs tid 405 v 599) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:19.785679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 prepare_pg_stats pg_stats(4 pgs tid 405 v 599) v1 from osd.0 2013-04-03 10:34:19.785685 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 got osd.0 osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:34:19.825328 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:19.825335 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:34:19.825359 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 preprocess_query pg_stats(3 pgs tid 525 v 599) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:19.825366 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 prepare_update pg_stats(3 pgs tid 525 v 599) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:19.825369 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 prepare_pg_stats pg_stats(3 pgs tid 525 v 599) v1 from osd.2 2013-04-03 10:34:19.825372 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[])) 2013-04-03 10:34:19.835875 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28970 encode_pending v 28971 2013-04-03 10:34:19.886394 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28970 update_from_paxos applying incremental 28971 2013-04-03 10:34:19.886424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 v28971: 240 pgs: 240 active+clean; 256 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 823B/s wr, 1op/s 2013-04-03 10:34:19.886444 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 send_pg_creates to 0 pgs 2013-04-03 10:34:19.886448 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 update_logger 2013-04-03 10:34:19.886459 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 create_pending v 28972 2013-04-03 10:34:19.886468 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28971 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:19.886486 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28971 _updated_stats for osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:19.886513 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 check_osd_map already seen 599 >= 599 2013-04-03 10:34:19.886517 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 update_logger 2013-04-03 10:34:19.886527 7fcdffcdb700 0 log [INF] : pgmap v28971: 240 pgs: 240 active+clean; 256 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 823B/s wr, 1op/s 2013-04-03 10:34:19.886593 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos 2013-04-03 10:34:19.886612 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos version 393976 summary v 393976 2013-04-03 10:34:19.886630 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:19.886672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos 2013-04-03 10:34:19.886686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos version 393976 summary v 393976 2013-04-03 10:34:19.886695 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:19.886705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:34:19.886727 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:19.886739 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:34:19.886748 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 logging 2013-04-03 10:34:16.935543 mon.0 10.252.24.25:6789/0 28 : [INF] mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:34:19.886766 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 logging 2013-04-03 10:34:19.886529 mon.0 10.252.24.25:6789/0 29 : [INF] pgmap v28971: 240 pgs: 240 active+clean; 256 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 823B/s wr, 1op/s 2013-04-03 10:34:19.911988 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos 2013-04-03 10:34:19.912003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos version 393976 summary v 393976 2013-04-03 10:34:19.912017 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:20.089238 7fcdfe8d5700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2116a00 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:32939/0) 2013-04-03 10:34:20.554314 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:20.554323 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 2013-04-03 10:34:20.554374 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 preprocess_query pg_stats(7 pgs tid 5 v 599) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:20.554383 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 prepare_update pg_stats(7 pgs tid 5 v 599) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:20.554386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 prepare_pg_stats pg_stats(7 pgs tid 5 v 599) v1 from osd.1 2013-04-03 10:34:20.554392 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 got osd.1 osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1113 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:34:20.722242 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:20.722252 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad900 from 10.252.24.88:6789/0 2013-04-03 10:34:20.722302 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 preprocess_query pg_stats(8 pgs tid 30433 v 599) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:20.722309 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 prepare_update pg_stats(8 pgs tid 30433 v 599) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:20.722313 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 prepare_pg_stats pg_stats(8 pgs tid 30433 v 599) v1 from osd.3 2013-04-03 10:34:20.722318 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 got osd.3 osd_stat(1192 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1191 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:34:20.886141 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393976 encode_pending v393977 2013-04-03 10:34:20.887417 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28971 encode_pending v 28972 2013-04-03 10:34:20.920775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos 2013-04-03 10:34:20.920812 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos version 393977 summary v 393976 2013-04-03 10:34:20.920835 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos latest full 393798 2013-04-03 10:34:20.920854 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393977 update_from_paxos applying incremental log 393977 2013-04-03 10:34:16.935543 mon.0 10.252.24.25:6789/0 28 : [INF] mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:34:20.920880 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393977 update_from_paxos applying incremental log 393977 2013-04-03 10:34:19.886529 mon.0 10.252.24.25:6789/0 29 : [INF] pgmap v28971: 240 pgs: 240 active+clean; 256 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 823B/s wr, 1op/s 2013-04-03 10:34:20.920921 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 check_subs 2013-04-03 10:34:20.920950 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 create_pending v 393978 2013-04-03 10:34:20.920971 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393977 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:20.972226 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28971 update_from_paxos applying incremental 28972 2013-04-03 10:34:20.972275 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 v28972: 240 pgs: 240 active+clean; 252 MB data, 5675 MB used, 1067 GB / 1079 GB avail; 210B/s rd, 33079B/s wr, 13op/s 2013-04-03 10:34:20.972297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 send_pg_creates to 0 pgs 2013-04-03 10:34:20.972300 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 update_logger 2013-04-03 10:34:20.972310 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 create_pending v 28973 2013-04-03 10:34:20.972319 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28972 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:20.972335 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28972 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:20.972377 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 check_osd_map already seen 599 >= 599 2013-04-03 10:34:20.972380 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 update_logger 2013-04-03 10:34:20.972392 7fcdffcdb700 0 log [INF] : pgmap v28972: 240 pgs: 240 active+clean; 252 MB data, 5675 MB used, 1067 GB / 1079 GB avail; 210B/s rd, 33079B/s wr, 13op/s 2013-04-03 10:34:20.972451 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos 2013-04-03 10:34:20.972467 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos version 393977 summary v 393977 2013-04-03 10:34:20.972487 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:20.972528 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos 2013-04-03 10:34:20.972545 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos version 393977 summary v 393977 2013-04-03 10:34:20.972554 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:20.972564 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:20.972580 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:20.972592 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:20.972601 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 logging 2013-04-03 10:34:20.972394 mon.0 10.252.24.25:6789/0 30 : [INF] pgmap v28972: 240 pgs: 240 active+clean; 252 MB data, 5675 MB used, 1067 GB / 1079 GB avail; 210B/s rd, 33079B/s wr, 13op/s 2013-04-03 10:34:20.997512 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos 2013-04-03 10:34:20.997527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos version 393977 summary v 393977 2013-04-03 10:34:20.997543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:20.997588 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:20.997595 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4600 from 10.252.24.88:6789/0 2013-04-03 10:34:20.997620 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 13 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:21.376394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21669 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:34:21.971308 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393977 encode_pending v393978 2013-04-03 10:34:22.114907 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos 2013-04-03 10:34:22.114943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393977 2013-04-03 10:34:22.114957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos latest full 393798 2013-04-03 10:34:22.114974 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393978 update_from_paxos applying incremental log 393978 2013-04-03 10:34:20.972394 mon.0 10.252.24.25:6789/0 30 : [INF] pgmap v28972: 240 pgs: 240 active+clean; 252 MB data, 5675 MB used, 1067 GB / 1079 GB avail; 210B/s rd, 33079B/s wr, 13op/s 2013-04-03 10:34:22.115029 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 check_subs 2013-04-03 10:34:22.115062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 create_pending v 393979 2013-04-03 10:34:22.115083 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393978 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:22.115144 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos 2013-04-03 10:34:22.115159 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 2013-04-03 10:34:22.115176 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:22.140010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos 2013-04-03 10:34:22.140026 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 2013-04-03 10:34:22.140044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:23.322564 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28972 check_down_pgs 2013-04-03 10:34:23.322597 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28972 v28972: 240 pgs: 240 active+clean; 252 MB data, 5675 MB used, 1067 GB / 1079 GB avail; 210B/s rd, 33079B/s wr, 13op/s 2013-04-03 10:34:23.322624 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:34:23.322651 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in 2013-04-03 10:34:23.322688 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 2013-04-03 10:34:23.322717 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos 2013-04-03 10:34:23.322735 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 2013-04-03 10:34:23.322744 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393978 log 2013-04-03 10:34:23.322760 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:23.322778 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:34:23.576306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140299 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:34:24.922091 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:24.922100 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4b80 from 10.252.24.88:6789/0 2013-04-03 10:34:24.922150 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 14 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:25.115804 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos 2013-04-03 10:34:25.115822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 2013-04-03 10:34:25.115839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:25.376497 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21670 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:34:25.554216 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:25.554224 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 2013-04-03 10:34:25.554258 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 preprocess_query pg_stats(1 pgs tid 6 v 599) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:25.554266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 prepare_update pg_stats(1 pgs tid 6 v 599) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:25.554270 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 prepare_pg_stats pg_stats(1 pgs tid 6 v 599) v1 from osd.1 2013-04-03 10:34:25.554275 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 got osd.1 osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:34:25.604453 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28972 encode_pending v 28973 2013-04-03 10:34:25.761415 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28972 update_from_paxos applying incremental 28973 2013-04-03 10:34:25.761435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 v28973: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 177B/s rd, 64792B/s wr, 27op/s 2013-04-03 10:34:25.761453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 send_pg_creates to 0 pgs 2013-04-03 10:34:25.761456 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 update_logger 2013-04-03 10:34:25.761465 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 create_pending v 28974 2013-04-03 10:34:25.761474 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28973 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:25.761492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 check_osd_map already seen 599 >= 599 2013-04-03 10:34:25.761495 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 update_logger 2013-04-03 10:34:25.761507 7fcdffcdb700 0 log [INF] : pgmap v28973: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 177B/s rd, 64792B/s wr, 27op/s 2013-04-03 10:34:25.761576 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos 2013-04-03 10:34:25.761593 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 2013-04-03 10:34:25.761610 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:25.761650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos 2013-04-03 10:34:25.761664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 2013-04-03 10:34:25.761673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:25.761683 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:25.761700 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:25.761711 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:25.761721 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:25.761508 mon.0 10.252.24.25:6789/0 31 : [INF] pgmap v28973: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 177B/s rd, 64792B/s wr, 27op/s 2013-04-03 10:34:25.773210 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos 2013-04-03 10:34:25.773225 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 2013-04-03 10:34:25.773239 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:25.773508 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:25.773516 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad900 from 10.252.24.88:6789/0 2013-04-03 10:34:25.773541 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 preprocess_query pg_stats(1 pgs tid 30434 v 599) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:25.773560 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 prepare_update pg_stats(1 pgs tid 30434 v 599) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:25.773565 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 prepare_pg_stats pg_stats(1 pgs tid 30434 v 599) v1 from osd.3 2013-04-03 10:34:25.773570 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 got osd.3 osd_stat(1192 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1192 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:34:25.773893 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:25.773898 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1c75200 from 10.252.24.88:6789/0 2013-04-03 10:34:25.773916 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos 2013-04-03 10:34:25.773931 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 2013-04-03 10:34:25.773940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 preprocess_query log(10 entries) v1 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:25.773950 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 preprocess_log log(10 entries) v1 from mds.0 2013-04-03 10:34:25.773995 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 prepare_update log(10 entries) v1 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:25.774006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 prepare_log log(10 entries) v1 from mds.0 2013-04-03 10:34:25.774016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.833513 mds.0 10.252.24.25:6802/3144 24 : [DBG] reconnect by client.11038 10.252.24.25:0/331381212 after 0.000310 2013-04-03 10:34:25.774034 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.833818 mds.0 10.252.24.25:6802/3144 25 : [DBG] reconnect by client.11042 10.252.24.25:0/2614196857 after 0.000622 2013-04-03 10:34:25.774052 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.835300 mds.0 10.252.24.25:6802/3144 26 : [DBG] reconnect by client.11041 10.252.24.25:0/3745741276 after 0.002096 2013-04-03 10:34:25.774069 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.835370 mds.0 10.252.24.25:6802/3144 27 : [DBG] reconnect by client.11074 10.252.24.32:0/3871608469 after 0.002174 2013-04-03 10:34:25.774088 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.835418 mds.0 10.252.24.25:6802/3144 28 : [DBG] reconnect by client.11054 10.252.24.30:0/2446411061 after 0.002224 2013-04-03 10:34:25.774106 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.835463 mds.0 10.252.24.25:6802/3144 29 : [DBG] reconnect by client.11045 10.252.24.25:0/2151250553 after 0.002268 2013-04-03 10:34:25.774124 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.835663 mds.0 10.252.24.25:6802/3144 30 : [DBG] reconnect by client.11802 10.252.24.30:0/517 after 0.002465 2013-04-03 10:34:25.774142 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.835708 mds.0 10.252.24.25:6802/3144 31 : [DBG] reconnect by client.11071 10.252.24.32:0/1396772501 after 0.002514 2013-04-03 10:34:25.774161 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.835801 mds.0 10.252.24.25:6802/3144 32 : [DBG] reconnect by client.11051 10.252.24.30:0/802047285 after 0.002603 2013-04-03 10:34:25.774179 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.859400 mds.0 10.252.24.25:6802/3144 33 : [DBG] reconnect by client.11228 10.252.24.25:0/1941 after 0.026202 2013-04-03 10:34:26.761168 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393978 encode_pending v393979 2013-04-03 10:34:26.762586 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28973 encode_pending v 28974 2013-04-03 10:34:26.800311 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos 2013-04-03 10:34:26.800336 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos version 393979 summary v 393978 2013-04-03 10:34:26.800349 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos latest full 393798 2013-04-03 10:34:26.800368 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.833513 mds.0 10.252.24.25:6802/3144 24 : [DBG] reconnect by client.11038 10.252.24.25:0/331381212 after 0.000310 2013-04-03 10:34:26.800394 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.833818 mds.0 10.252.24.25:6802/3144 25 : [DBG] reconnect by client.11042 10.252.24.25:0/2614196857 after 0.000622 2013-04-03 10:34:26.800413 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.835300 mds.0 10.252.24.25:6802/3144 26 : [DBG] reconnect by client.11041 10.252.24.25:0/3745741276 after 0.002096 2013-04-03 10:34:26.800432 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.835370 mds.0 10.252.24.25:6802/3144 27 : [DBG] reconnect by client.11074 10.252.24.32:0/3871608469 after 0.002174 2013-04-03 10:34:26.800449 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.835418 mds.0 10.252.24.25:6802/3144 28 : [DBG] reconnect by client.11054 10.252.24.30:0/2446411061 after 0.002224 2013-04-03 10:34:26.800467 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.835463 mds.0 10.252.24.25:6802/3144 29 : [DBG] reconnect by client.11045 10.252.24.25:0/2151250553 after 0.002268 2013-04-03 10:34:26.800485 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.835663 mds.0 10.252.24.25:6802/3144 30 : [DBG] reconnect by client.11802 10.252.24.30:0/517 after 0.002465 2013-04-03 10:34:26.800510 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.835708 mds.0 10.252.24.25:6802/3144 31 : [DBG] reconnect by client.11071 10.252.24.32:0/1396772501 after 0.002514 2013-04-03 10:34:26.800531 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.835801 mds.0 10.252.24.25:6802/3144 32 : [DBG] reconnect by client.11051 10.252.24.30:0/802047285 after 0.002603 2013-04-03 10:34:26.800549 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.859400 mds.0 10.252.24.25:6802/3144 33 : [DBG] reconnect by client.11228 10.252.24.25:0/1941 after 0.026202 2013-04-03 10:34:26.800567 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:25.761508 mon.0 10.252.24.25:6789/0 31 : [INF] pgmap v28973: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 177B/s rd, 64792B/s wr, 27op/s 2013-04-03 10:34:26.800607 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 check_subs 2013-04-03 10:34:26.800636 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 create_pending v 393980 2013-04-03 10:34:26.800657 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:26.800677 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 _updated_log for mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:26.838237 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28973 update_from_paxos applying incremental 28974 2013-04-03 10:34:26.838257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 v28974: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 37124B/s wr, 16op/s 2013-04-03 10:34:26.838276 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 send_pg_creates to 0 pgs 2013-04-03 10:34:26.838280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 update_logger 2013-04-03 10:34:26.838290 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 create_pending v 28975 2013-04-03 10:34:26.838299 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28974 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:26.838315 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 check_osd_map already seen 599 >= 599 2013-04-03 10:34:26.838317 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 update_logger 2013-04-03 10:34:26.838327 7fcdffcdb700 0 log [INF] : pgmap v28974: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 37124B/s wr, 16op/s 2013-04-03 10:34:26.838381 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos 2013-04-03 10:34:26.838397 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos version 393979 summary v 393979 2013-04-03 10:34:26.838415 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:26.838456 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos 2013-04-03 10:34:26.838470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos version 393979 summary v 393979 2013-04-03 10:34:26.838482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:26.838494 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:26.838509 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:26.838521 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:26.838532 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 logging 2013-04-03 10:34:26.838328 mon.0 10.252.24.25:6789/0 32 : [INF] pgmap v28974: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 37124B/s wr, 16op/s 2013-04-03 10:34:26.862957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos 2013-04-03 10:34:26.862972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos version 393979 summary v 393979 2013-04-03 10:34:26.862986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:27.576407 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140300 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:34:27.838032 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393979 encode_pending v393980 2013-04-03 10:34:27.870706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos 2013-04-03 10:34:27.870749 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos version 393980 summary v 393979 2013-04-03 10:34:27.870763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos latest full 393798 2013-04-03 10:34:27.870782 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393980 update_from_paxos applying incremental log 393980 2013-04-03 10:34:26.838328 mon.0 10.252.24.25:6789/0 32 : [INF] pgmap v28974: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 37124B/s wr, 16op/s 2013-04-03 10:34:27.870829 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 check_subs 2013-04-03 10:34:27.870858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 create_pending v 393981 2013-04-03 10:34:27.870879 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393980 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:27.870939 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos 2013-04-03 10:34:27.870954 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos version 393980 summary v 393980 2013-04-03 10:34:27.870972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:27.888032 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos 2013-04-03 10:34:27.888048 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos version 393980 summary v 393980 2013-04-03 10:34:27.888062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:28.322920 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28974 check_down_pgs 2013-04-03 10:34:28.322956 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28974 v28974: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 37124B/s wr, 16op/s 2013-04-03 10:34:28.322982 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:34:28.323005 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in 2013-04-03 10:34:28.323039 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 2013-04-03 10:34:28.323068 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos 2013-04-03 10:34:28.323085 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos version 393980 summary v 393980 2013-04-03 10:34:28.323098 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393980 log 2013-04-03 10:34:28.323114 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:28.323132 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:34:28.922193 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:28.922202 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2145dc0 from 10.252.24.88:6789/0 2013-04-03 10:34:28.922253 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 15 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:29.376628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21671 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:34:29.826516 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:29.826526 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad900 from 10.252.24.88:6789/0 2013-04-03 10:34:29.826574 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 preprocess_query pg_stats(1 pgs tid 526 v 599) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:29.826596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 prepare_update pg_stats(1 pgs tid 526 v 599) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:29.826602 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 prepare_pg_stats pg_stats(1 pgs tid 526 v 599) v1 from osd.2 2013-04-03 10:34:29.826608 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[])) 2013-04-03 10:34:29.876789 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28974 encode_pending v 28975 2013-04-03 10:34:29.913106 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28974 update_from_paxos applying incremental 28975 2013-04-03 10:34:29.913121 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 v28975: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2157B/s wr, 0op/s 2013-04-03 10:34:29.913140 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 send_pg_creates to 0 pgs 2013-04-03 10:34:29.913144 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 update_logger 2013-04-03 10:34:29.913153 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 create_pending v 28976 2013-04-03 10:34:29.913162 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28975 _updated_stats for osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:29.913182 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 check_osd_map already seen 599 >= 599 2013-04-03 10:34:29.913185 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 update_logger 2013-04-03 10:34:29.913197 7fcdffcdb700 0 log [INF] : pgmap v28975: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2157B/s wr, 0op/s 2013-04-03 10:34:29.913255 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos 2013-04-03 10:34:29.913273 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos version 393980 summary v 393980 2013-04-03 10:34:29.913291 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:29.913342 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos 2013-04-03 10:34:29.913356 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos version 393980 summary v 393980 2013-04-03 10:34:29.913365 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:29.913375 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:29.913391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:29.913400 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:29.913409 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 logging 2013-04-03 10:34:29.913198 mon.0 10.252.24.25:6789/0 33 : [INF] pgmap v28975: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2157B/s wr, 0op/s 2013-04-03 10:34:29.938232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos 2013-04-03 10:34:29.938248 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos version 393980 summary v 393980 2013-04-03 10:34:29.938262 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:30.554739 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:30.554749 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:34:30.554800 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 preprocess_query pg_stats(1 pgs tid 7 v 599) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:30.554822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 prepare_update pg_stats(1 pgs tid 7 v 599) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:30.554828 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 prepare_pg_stats pg_stats(1 pgs tid 7 v 599) v1 from osd.1 2013-04-03 10:34:30.554834 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 got osd.1 osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:34:30.912328 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393980 encode_pending v393981 2013-04-03 10:34:30.913543 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28975 encode_pending v 28976 2013-04-03 10:34:30.963635 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos 2013-04-03 10:34:30.963660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos version 393981 summary v 393980 2013-04-03 10:34:30.963679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos latest full 393798 2013-04-03 10:34:30.963698 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393981 update_from_paxos applying incremental log 393981 2013-04-03 10:34:29.913198 mon.0 10.252.24.25:6789/0 33 : [INF] pgmap v28975: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2157B/s wr, 0op/s 2013-04-03 10:34:30.963750 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 check_subs 2013-04-03 10:34:30.963781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 create_pending v 393982 2013-04-03 10:34:30.963802 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393981 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:30.991321 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28975 update_from_paxos applying incremental 28976 2013-04-03 10:34:30.991342 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 v28976: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2220B/s wr, 0op/s 2013-04-03 10:34:30.991362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 send_pg_creates to 0 pgs 2013-04-03 10:34:30.991366 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 update_logger 2013-04-03 10:34:30.991376 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 create_pending v 28977 2013-04-03 10:34:30.991385 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28976 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:30.991407 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 check_osd_map already seen 599 >= 599 2013-04-03 10:34:30.991410 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 update_logger 2013-04-03 10:34:30.991422 7fcdffcdb700 0 log [INF] : pgmap v28976: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2220B/s wr, 0op/s 2013-04-03 10:34:30.991478 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos 2013-04-03 10:34:30.991496 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos version 393981 summary v 393981 2013-04-03 10:34:30.991514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:30.991555 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos 2013-04-03 10:34:30.991570 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos version 393981 summary v 393981 2013-04-03 10:34:30.991578 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:30.991590 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:30.991607 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:30.991619 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:30.991630 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 logging 2013-04-03 10:34:30.991423 mon.0 10.252.24.25:6789/0 34 : [INF] pgmap v28976: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2220B/s wr, 0op/s 2013-04-03 10:34:31.002943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos 2013-04-03 10:34:31.002961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos version 393981 summary v 393981 2013-04-03 10:34:31.002976 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:31.576551 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140301 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:34:31.991089 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393981 encode_pending v393982 2013-04-03 10:34:32.019598 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos 2013-04-03 10:34:32.019625 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos version 393982 summary v 393981 2013-04-03 10:34:32.019639 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos latest full 393798 2013-04-03 10:34:32.019657 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393982 update_from_paxos applying incremental log 393982 2013-04-03 10:34:30.991423 mon.0 10.252.24.25:6789/0 34 : [INF] pgmap v28976: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2220B/s wr, 0op/s 2013-04-03 10:34:32.019698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 check_subs 2013-04-03 10:34:32.019727 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 create_pending v 393983 2013-04-03 10:34:32.019749 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393982 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:32.020012 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos 2013-04-03 10:34:32.020031 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos version 393982 summary v 393982 2013-04-03 10:34:32.020051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:32.044022 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos 2013-04-03 10:34:32.044039 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos version 393982 summary v 393982 2013-04-03 10:34:32.044054 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:32.922346 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:32.922355 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e3b00 from 10.252.24.88:6789/0 2013-04-03 10:34:32.922406 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 16 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:33.323254 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28976 check_down_pgs 2013-04-03 10:34:33.323292 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28976 v28976: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2220B/s wr, 0op/s 2013-04-03 10:34:33.323318 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:34:33.323340 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in 2013-04-03 10:34:33.323376 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 2013-04-03 10:34:33.323404 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos 2013-04-03 10:34:33.323421 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos version 393982 summary v 393982 2013-04-03 10:34:33.323430 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393982 log 2013-04-03 10:34:33.323446 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:33.323464 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:34:33.376701 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21672 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:34:34.826890 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:34.826900 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:34:34.826952 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 preprocess_query pg_stats(1 pgs tid 527 v 599) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:34.826974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 prepare_update pg_stats(1 pgs tid 527 v 599) v1 from osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:34.826980 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 prepare_pg_stats pg_stats(1 pgs tid 527 v 599) v1 from osd.2 2013-04-03 10:34:34.826986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[])) 2013-04-03 10:34:34.877161 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28976 encode_pending v 28977 2013-04-03 10:34:34.907965 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28976 update_from_paxos applying incremental 28977 2013-04-03 10:34:34.907984 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 v28977: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s 2013-04-03 10:34:34.908002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 send_pg_creates to 0 pgs 2013-04-03 10:34:34.908004 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 update_logger 2013-04-03 10:34:34.908014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 create_pending v 28978 2013-04-03 10:34:34.908023 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28977 _updated_stats for osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:34.908042 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 check_osd_map already seen 599 >= 599 2013-04-03 10:34:34.908046 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 update_logger 2013-04-03 10:34:34.908057 7fcdffcdb700 0 log [INF] : pgmap v28977: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s 2013-04-03 10:34:34.908468 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos 2013-04-03 10:34:34.908484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos version 393982 summary v 393982 2013-04-03 10:34:34.908493 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:34.908504 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:34.908520 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:34.908531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:34.908540 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 logging 2013-04-03 10:34:34.908058 mon.0 10.252.24.25:6789/0 35 : [INF] pgmap v28977: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s 2013-04-03 10:34:34.946197 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos 2013-04-03 10:34:34.946213 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos version 393982 summary v 393982 2013-04-03 10:34:34.946233 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:34.957706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos 2013-04-03 10:34:34.957722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos version 393982 summary v 393982 2013-04-03 10:34:34.957736 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:35.555262 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:35.555270 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad900 from 10.252.24.88:6789/0 2013-04-03 10:34:35.555506 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 preprocess_query pg_stats(1 pgs tid 8 v 599) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:35.555526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 prepare_update pg_stats(1 pgs tid 8 v 599) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:35.555532 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 prepare_pg_stats pg_stats(1 pgs tid 8 v 599) v1 from osd.1 2013-04-03 10:34:35.555537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 got osd.1 osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:34:35.576646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140302 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:34:35.907636 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393982 encode_pending v393983 2013-04-03 10:34:35.932340 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos 2013-04-03 10:34:35.932366 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos version 393983 summary v 393982 2013-04-03 10:34:35.932385 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos latest full 393798 2013-04-03 10:34:35.932403 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393983 update_from_paxos applying incremental log 393983 2013-04-03 10:34:34.908058 mon.0 10.252.24.25:6789/0 35 : [INF] pgmap v28977: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s 2013-04-03 10:34:35.932450 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 check_subs 2013-04-03 10:34:35.932480 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 create_pending v 393984 2013-04-03 10:34:35.932501 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393983 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:35.932562 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos 2013-04-03 10:34:35.932577 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos version 393983 summary v 393983 2013-04-03 10:34:35.932594 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:35.943876 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos 2013-04-03 10:34:35.943892 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos version 393983 summary v 393983 2013-04-03 10:34:35.943907 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:35.945928 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28977 encode_pending v 28978 2013-04-03 10:34:35.977781 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28977 update_from_paxos applying incremental 28978 2013-04-03 10:34:35.977800 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28978 v28978: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 7732B/s wr, 2op/s 2013-04-03 10:34:35.977827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28978 send_pg_creates to 0 pgs 2013-04-03 10:34:35.977831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28978 update_logger 2013-04-03 10:34:35.977841 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28978 create_pending v 28979 2013-04-03 10:34:35.977849 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28978 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:35.977865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28978 check_osd_map already seen 599 >= 599 2013-04-03 10:34:35.977868 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28978 update_logger 2013-04-03 10:34:35.977880 7fcdffcdb700 0 log [INF] : pgmap v28978: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 7732B/s wr, 2op/s 2013-04-03 10:34:35.977932 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos 2013-04-03 10:34:35.977948 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos version 393983 summary v 393983 2013-04-03 10:34:35.977964 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:35.978002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos 2013-04-03 10:34:35.978019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos version 393983 summary v 393983 2013-04-03 10:34:35.978028 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:35.978039 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:35.978053 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:35.978065 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:35.978074 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 logging 2013-04-03 10:34:35.977881 mon.0 10.252.24.25:6789/0 36 : [INF] pgmap v28978: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 7732B/s wr, 2op/s 2013-04-03 10:34:35.990683 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos 2013-04-03 10:34:35.990702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos version 393983 summary v 393983 2013-04-03 10:34:35.990718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:36.922585 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:36.922595 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae600 from 10.252.24.88:6789/0 2013-04-03 10:34:36.922645 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 17 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:36.977541 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393983 encode_pending v393984 2013-04-03 10:34:37.140559 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 update_from_paxos 2013-04-03 10:34:37.140584 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 update_from_paxos version 393984 summary v 393983 2013-04-03 10:34:37.140598 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 update_from_paxos latest full 393798 2013-04-03 10:34:37.140616 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393984 update_from_paxos applying incremental log 393984 2013-04-03 10:34:35.977881 mon.0 10.252.24.25:6789/0 36 : [INF] pgmap v28978: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 7732B/s wr, 2op/s 2013-04-03 10:34:37.140660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 check_subs 2013-04-03 10:34:37.140689 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 create_pending v 393985 2013-04-03 10:34:37.140710 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393984 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:37.140769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 update_from_paxos 2013-04-03 10:34:37.140784 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 update_from_paxos version 393984 summary v 393984 2013-04-03 10:34:37.140804 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:37.164940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 update_from_paxos 2013-04-03 10:34:37.164956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 update_from_paxos version 393984 summary v 393984 2013-04-03 10:34:37.164972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:34:37.376825 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21673 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:34:37.810087 7fcdfe8d5700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.30:6789/0 mon protocol 2 2013-04-03 10:34:37.810202 7fcdfe8d5700 0 -- 10.252.24.25:6789/0 >> 10.252.24.30:6789/0 pipe(0x26dc500 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state connecting 2013-04-03 10:34:38.157276 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:34:38.157288 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:34:38.157299 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 2013-04-03 10:34:38.323598 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 check_down_pgs 2013-04-03 10:34:38.323627 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 v28978: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 7732B/s wr, 2op/s 2013-04-03 10:34:38.323658 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:34:38.323682 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 e599: 5 osds: 5 up, 5 in 2013-04-03 10:34:38.323719 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 min_last_epoch_clean 593 2013-04-03 10:34:38.323749 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos 2013-04-03 10:34:38.323767 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos version 393984 summary v 393984 2013-04-03 10:34:38.323776 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 log 2013-04-03 10:34:38.323793 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:38.323811 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:34:39.576812 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140303 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:34:40.121289 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x26dc280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:32948/0) 2013-04-03 10:34:40.141464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos 2013-04-03 10:34:40.141483 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos version 393984 summary v 393984 2013-04-03 10:34:40.141502 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:40.922616 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:40.922625 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273bb80 from 10.252.24.88:6789/0 2013-04-03 10:34:40.922661 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 18 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:41.376967 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21674 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:34:42.481949 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:42.481960 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2071800 from 10.252.24.88:6789/0 2013-04-03 10:34:42.482009 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 preprocess_query osd_boot(osd.2 booted 0 v599) v3 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:42.482018 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 preprocess_boot from osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:42.482023 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 prepare_update osd_boot(osd.2 booted 0 v599) v3 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:42.482027 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 prepare_boot from osd.2 10.252.24.30:6800/1508 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.2 eb9ff30e-cc75-49d3-8ef9-2f478e173371 e599 [98,599] lci=[593,599]) cluster_addr 10.252.24.30:6804/1508 hb_addr 10.252.24.30:6805/1508 2013-04-03 10:34:42.482038 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 prepare_boot was up, first marking down osd.2 10.252.24.30:6801/10373 2013-04-03 10:34:42.482042 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 should_propose 2013-04-03 10:34:42.532195 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 min_last_epoch_clean 593 2013-04-03 10:34:42.532253 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 min_last_epoch_clean 593 2013-04-03 10:34:42.532306 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 encode_pending e 600 2013-04-03 10:34:42.532311 7fce004dc700 2 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 osd.2 DOWN 2013-04-03 10:34:42.567888 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 update_from_paxos applying incremental 600 2013-04-03 10:34:42.567932 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 e600: 5 osds: 4 up, 5 in 2013-04-03 10:34:42.568912 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 adding osd.2 to down_pending_out map 2013-04-03 10:34:42.568965 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 check_osd_map applying osdmap e600 to pg_map 2013-04-03 10:34:42.568984 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 register_new_pgs checking pg pools for osdmap epoch 600, last_pg_scan 341 2013-04-03 10:34:42.568989 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 2013-04-03 10:34:42.568993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:34:42.568996 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:34:42.568999 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 2013-04-03 10:34:42.569002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 2013-04-03 10:34:42.569005 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 2013-04-03 10:34:42.569007 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 2013-04-03 10:34:42.569010 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 2013-04-03 10:34:42.569013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 2013-04-03 10:34:42.569023 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs 2013-04-03 10:34:42.569025 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 check_down_pgs 2013-04-03 10:34:42.569027 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.d stale with acting [2,0] 2013-04-03 10:34:42.569034 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 8.3 stale with acting [2,3] 2013-04-03 10:34:42.569037 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.9 stale with acting [2,3] 2013-04-03 10:34:42.569041 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.a stale with acting [2,3] 2013-04-03 10:34:42.569044 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.b stale with acting [2,3] 2013-04-03 10:34:42.569047 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 7.0 stale with acting [2,3] 2013-04-03 10:34:42.569050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 6.1 stale with acting [2,3] 2013-04-03 10:34:42.569053 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 5.2 stale with acting [2,3] 2013-04-03 10:34:42.569055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 4.3 stale with acting [2,3] 2013-04-03 10:34:42.569058 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 3.4 stale with acting [2,3] 2013-04-03 10:34:42.569061 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.5 stale with acting [2,3] 2013-04-03 10:34:42.569064 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.6 stale with acting [2,3] 2013-04-03 10:34:42.569067 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.7 stale with acting [2,3] 2013-04-03 10:34:42.569070 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 5.0 stale with acting [2,0] 2013-04-03 10:34:42.569072 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 4.1 stale with acting [2,0] 2013-04-03 10:34:42.569075 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.4 stale with acting [2,0] 2013-04-03 10:34:42.569078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.5 stale with acting [2,0] 2013-04-03 10:34:42.569081 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 7.4 stale with acting [2,3] 2013-04-03 10:34:42.569084 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 6.5 stale with acting [2,3] 2013-04-03 10:34:42.569087 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 5.6 stale with acting [2,3] 2013-04-03 10:34:42.569091 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 4.7 stale with acting [2,3] 2013-04-03 10:34:42.569093 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.1 stale with acting [2,4] 2013-04-03 10:34:42.569096 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.2 stale with acting [2,4] 2013-04-03 10:34:42.569099 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.3 stale with acting [2,4] 2013-04-03 10:34:42.569102 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 3.0 stale with acting [2,4] 2013-04-03 10:34:42.569105 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 3.2 stale with acting [2,0] 2013-04-03 10:34:42.569113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.3 stale with acting [2,0] 2013-04-03 10:34:42.569116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 7.7 stale with acting [2,0] 2013-04-03 10:34:42.569120 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.3c stale with acting [2,3] 2013-04-03 10:34:42.569123 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.3e stale with acting [2,3] 2013-04-03 10:34:42.569126 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.3d stale with acting [2,3] 2013-04-03 10:34:42.569129 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.39 stale with acting [2,0] 2013-04-03 10:34:42.569132 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.3a stale with acting [2,0] 2013-04-03 10:34:42.569135 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.3b stale with acting [2,0] 2013-04-03 10:34:42.569138 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.38 stale with acting [2,3] 2013-04-03 10:34:42.569141 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.3a stale with acting [2,3] 2013-04-03 10:34:42.569144 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.39 stale with acting [2,3] 2013-04-03 10:34:42.569147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.34 stale with acting [2,1] 2013-04-03 10:34:42.569150 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.36 stale with acting [2,1] 2013-04-03 10:34:42.569156 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.35 stale with acting [2,1] 2013-04-03 10:34:42.569159 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.29 stale with acting [2,0] 2013-04-03 10:34:42.569162 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.2a stale with acting [2,0] 2013-04-03 10:34:42.569165 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.2b stale with acting [2,0] 2013-04-03 10:34:42.569168 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.28 stale with acting [2,0] 2013-04-03 10:34:42.569171 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.2a stale with acting [2,0] 2013-04-03 10:34:42.569173 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.28 stale with acting [2,1] 2013-04-03 10:34:42.569176 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.29 stale with acting [2,1] 2013-04-03 10:34:42.569179 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.29 stale with acting [2,0] 2013-04-03 10:34:42.569182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.27 stale with acting [2,1] 2013-04-03 10:34:42.569185 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.21 stale with acting [2,3] 2013-04-03 10:34:42.569188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.22 stale with acting [2,3] 2013-04-03 10:34:42.569191 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.23 stale with acting [2,3] 2013-04-03 10:34:42.569194 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.20 stale with acting [2,1] 2013-04-03 10:34:42.569197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.22 stale with acting [2,1] 2013-04-03 10:34:42.569199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.21 stale with acting [2,1] 2013-04-03 10:34:42.569203 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.1c stale with acting [2,3] 2013-04-03 10:34:42.569205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.1b stale with acting [2,3] 2013-04-03 10:34:42.569208 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.19 stale with acting [2,3] 2013-04-03 10:34:42.569214 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.18 stale with acting [2,3] 2013-04-03 10:34:42.569217 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.18 stale with acting [2,3] 2013-04-03 10:34:42.569220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.1a stale with acting [2,3] 2013-04-03 10:34:42.569223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.17 stale with acting [2,3] 2013-04-03 10:34:42.569229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.17 stale with acting [2,3] 2013-04-03 10:34:42.569232 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.16 stale with acting [2,3] 2013-04-03 10:34:42.569235 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.f stale with acting [2,3] 2013-04-03 10:34:42.569238 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.e stale with acting [2,3] 2013-04-03 10:34:42.569241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.d stale with acting [2,3] 2013-04-03 10:34:42.569244 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 8.7 stale with acting [2,3] 2013-04-03 10:34:42.569247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.e stale with acting [2,0] 2013-04-03 10:34:42.569249 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.c stale with acting [2,0] 2013-04-03 10:34:42.569252 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 8.6 stale with acting [2,0] 2013-04-03 10:34:42.569305 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 encode_pending v 28979 2013-04-03 10:34:42.571087 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 send_pg_creates to 0 pgs 2013-04-03 10:34:42.571093 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 send_to_waiting 600 2013-04-03 10:34:42.571100 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 share_map_with_random_osd no up osd on our session map 2013-04-03 10:34:42.571102 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 update_logger 2013-04-03 10:34:42.571122 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 create_pending e 601 2013-04-03 10:34:42.571125 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 remove_redundant_pg_temp 2013-04-03 10:34:42.571127 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 remove_down_pg_temp 2013-04-03 10:34:42.571160 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_query osd_boot(osd.2 booted 0 v599) v3 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:42.571169 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_boot from osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:42.571173 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_update osd_boot(osd.2 booted 0 v599) v3 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:42.571176 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_boot from osd.2 10.252.24.30:6800/1508 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.2 eb9ff30e-cc75-49d3-8ef9-2f478e173371 e599 [98,599] lci=[593,599]) cluster_addr 10.252.24.30:6804/1508 hb_addr 10.252.24.30:6805/1508 2013-04-03 10:34:42.571188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 setting osd.2 uuid to eb9ff30e-cc75-49d3-8ef9-2f478e173371 2013-04-03 10:34:42.571192 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 old osd_info: up_from 593 up_thru 598 down_at 600 last_clean_interval [586,590) lost_at 95 2013-04-03 10:34:42.571195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_boot osd.2 last_clean_interval [586,590) -> [593-599) 2013-04-03 10:34:42.571198 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 not laggy, new xi down_stamp 2013-04-03 10:34:42.532310 laggy_probability 0 laggy_interval 0 2013-04-03 10:34:42.571208 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 should_propose 2013-04-03 10:34:42.571220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 update_logger 2013-04-03 10:34:42.571227 7fcdffcdb700 0 log [INF] : osdmap e600: 5 osds: 4 up, 5 in 2013-04-03 10:34:42.571295 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos 2013-04-03 10:34:42.571311 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos version 393984 summary v 393984 2013-04-03 10:34:42.571321 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:42.571333 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:42.571351 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:42.571364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:42.571376 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 logging 2013-04-03 10:34:42.571227 mon.0 10.252.24.25:6789/0 37 : [INF] osdmap e600: 5 osds: 4 up, 5 in 2013-04-03 10:34:42.690096 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 update_from_paxos applying incremental 28979 2013-04-03 10:34:42.690270 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 v28979: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 4526B/s wr, 1op/s 2013-04-03 10:34:42.690300 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 send_pg_creates to 0 pgs 2013-04-03 10:34:42.690304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 update_logger 2013-04-03 10:34:42.690323 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 create_pending v 28980 2013-04-03 10:34:42.690334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 check_osd_map already seen 600 >= 600 2013-04-03 10:34:42.690339 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 update_logger 2013-04-03 10:34:42.690350 7fcdffcdb700 0 log [INF] : pgmap v28979: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 4526B/s wr, 1op/s 2013-04-03 10:34:42.690404 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos 2013-04-03 10:34:42.690421 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos version 393984 summary v 393984 2013-04-03 10:34:42.690439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:42.690482 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos 2013-04-03 10:34:42.690497 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos version 393984 summary v 393984 2013-04-03 10:34:42.690507 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:42.690518 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:42.690533 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 nothing new 2013-04-03 10:34:42.766751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos 2013-04-03 10:34:42.766768 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos version 393984 summary v 393984 2013-04-03 10:34:42.766784 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:42.766814 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:42.766822 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x22248c0 from 10.252.24.88:6789/0 2013-04-03 10:34:42.766845 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_query osd_alive(want up_thru 600 have 600) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:42.766852 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_alive want up_thru 600 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:42.766857 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_update osd_alive(want up_thru 600 have 600) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:42.766860 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_alive want up_thru 600 have 600 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:42.766864 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 should_propose 2013-04-03 10:34:42.766886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:42.766893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x22241c0 from 10.252.24.88:6789/0 2013-04-03 10:34:42.766915 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_query osd_alive(want up_thru 600 have 600) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:34:42.766921 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_alive want up_thru 600 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:34:42.766925 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_update osd_alive(want up_thru 600 have 600) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:34:42.766928 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_alive want up_thru 600 have 600 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:34:42.766931 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 should_propose 2013-04-03 10:34:42.766947 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:42.766952 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2224c40 from 10.252.24.88:6789/0 2013-04-03 10:34:42.766974 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_query osd_alive(want up_thru 600 have 600) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:42.766981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_alive want up_thru 600 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:42.766985 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_update osd_alive(want up_thru 600 have 600) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:42.766988 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_alive want up_thru 600 have 600 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:42.766991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 should_propose 2013-04-03 10:34:42.767007 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:42.767013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2224540 from 10.252.24.88:6789/0 2013-04-03 10:34:42.767033 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_query osd_alive(want up_thru 600 have 600) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:42.767040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_alive want up_thru 600 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:42.767044 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_update osd_alive(want up_thru 600 have 600) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:42.767047 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_alive want up_thru 600 have 600 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:42.767050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 should_propose 2013-04-03 10:34:43.323962 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 check_down_pgs 2013-04-03 10:34:43.323988 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 v28979: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 4526B/s wr, 1op/s 2013-04-03 10:34:43.324016 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:34:43.324039 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 e600: 5 osds: 4 up, 5 in 2013-04-03 10:34:43.324076 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 min_last_epoch_clean 593 2013-04-03 10:34:43.324106 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos 2013-04-03 10:34:43.324123 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos version 393984 summary v 393984 2013-04-03 10:34:43.324131 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 log 2013-04-03 10:34:43.324147 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:43.324166 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:34:43.567657 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 encode_pending v393985 2013-04-03 10:34:43.569299 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 min_last_epoch_clean 593 2013-04-03 10:34:43.569358 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 min_last_epoch_clean 593 2013-04-03 10:34:43.569407 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 encode_pending e 601 2013-04-03 10:34:43.569411 7fce004dc700 2 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 osd.2 UP 10.252.24.30:6800/1508 2013-04-03 10:34:43.576902 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140304 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:34:43.833997 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos 2013-04-03 10:34:43.834023 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos version 393985 summary v 393984 2013-04-03 10:34:43.834037 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos latest full 393798 2013-04-03 10:34:43.834056 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos applying incremental log 393985 2013-04-03 10:34:42.571227 mon.0 10.252.24.25:6789/0 37 : [INF] osdmap e600: 5 osds: 4 up, 5 in 2013-04-03 10:34:43.834097 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 check_subs 2013-04-03 10:34:43.834127 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 create_pending v 393986 2013-04-03 10:34:43.834149 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:43.871564 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 update_from_paxos applying incremental 601 2013-04-03 10:34:43.871612 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 e601: 5 osds: 5 up, 5 in 2013-04-03 10:34:43.872723 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 check_osd_map applying osdmap e601 to pg_map 2013-04-03 10:34:43.872751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 register_new_pgs checking pg pools for osdmap epoch 601, last_pg_scan 341 2013-04-03 10:34:43.872754 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 2013-04-03 10:34:43.872758 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:34:43.872762 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:34:43.872765 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 2013-04-03 10:34:43.872768 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 2013-04-03 10:34:43.872771 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 2013-04-03 10:34:43.872774 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 2013-04-03 10:34:43.872783 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 2013-04-03 10:34:43.872786 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 2013-04-03 10:34:43.872795 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs 2013-04-03 10:34:43.872844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 encode_pending v 28980 2013-04-03 10:34:43.873114 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 send_pg_creates to 0 pgs 2013-04-03 10:34:43.873128 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_to_waiting 601 2013-04-03 10:34:43.873133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 share_map_with_random_osd no up osd on our session map 2013-04-03 10:34:43.873140 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 update_logger 2013-04-03 10:34:43.873169 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 create_pending e 602 2013-04-03 10:34:43.873174 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 remove_redundant_pg_temp 2013-04-03 10:34:43.873176 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 remove_down_pg_temp 2013-04-03 10:34:43.873195 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 _booted osd.2 10.252.24.30:6800/1508 w 0 from 599 2013-04-03 10:34:43.873212 7fcdffcdb700 0 log [INF] : osd.2 10.252.24.30:6800/1508 boot 2013-04-03 10:34:43.873247 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_latest to osd.2 10.252.24.30:6800/1508 start 600 2013-04-03 10:34:43.873254 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_incremental [600..601] to osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:43.873266 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 build_incremental [600..601] 2013-04-03 10:34:43.873313 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 _reply_map 600 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:43.873331 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_latest to osd.1 10.252.24.25:6805/3572 start 600 2013-04-03 10:34:43.873337 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_incremental [600..601] to osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:43.873348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 build_incremental [600..601] 2013-04-03 10:34:43.873390 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 _reply_map 600 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:34:43.873409 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_latest to osd.4 10.252.24.32:6800/13473 start 600 2013-04-03 10:34:43.873414 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_incremental [600..601] to osd.4 10.252.24.32:6800/13473 2013-04-03 10:34:43.873425 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 build_incremental [600..601] 2013-04-03 10:34:43.873465 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 _reply_map 600 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:43.873489 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_latest to osd.3 10.252.24.88:6801/17538 start 600 2013-04-03 10:34:43.873495 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_incremental [600..601] to osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:43.873506 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 build_incremental [600..601] 2013-04-03 10:34:43.873548 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 _reply_map 600 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:43.873568 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_latest to osd.0 10.252.58.31:6802/2682 start 600 2013-04-03 10:34:43.873573 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_incremental [600..601] to osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:43.873584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 build_incremental [600..601] 2013-04-03 10:34:43.873626 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 update_logger 2013-04-03 10:34:43.873633 7fcdffcdb700 0 log [INF] : osdmap e601: 5 osds: 5 up, 5 in 2013-04-03 10:34:43.873694 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos 2013-04-03 10:34:43.873710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos version 393985 summary v 393985 2013-04-03 10:34:43.873719 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:43.873731 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:34:43.873749 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:43.873762 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:34:43.873773 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 logging 2013-04-03 10:34:42.690351 mon.0 10.252.24.25:6789/0 38 : [INF] pgmap v28979: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 4526B/s wr, 1op/s 2013-04-03 10:34:43.873791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 logging 2013-04-03 10:34:43.873213 mon.0 10.252.24.25:6789/0 39 : [INF] osd.2 10.252.24.30:6800/1508 boot 2013-04-03 10:34:43.873848 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos 2013-04-03 10:34:43.873863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos version 393985 summary v 393985 2013-04-03 10:34:43.873873 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:43.873884 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:43.873899 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 nothing new 2013-04-03 10:34:43.923198 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 update_from_paxos applying incremental 28980 2013-04-03 10:34:43.923207 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 v28980: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:43.923224 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 send_pg_creates to 0 pgs 2013-04-03 10:34:43.923226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 update_logger 2013-04-03 10:34:43.923235 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 create_pending v 28981 2013-04-03 10:34:43.923244 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 check_osd_map already seen 601 >= 601 2013-04-03 10:34:43.923247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 update_logger 2013-04-03 10:34:43.923256 7fcdffcdb700 0 log [INF] : pgmap v28980: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:43.923330 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos 2013-04-03 10:34:43.923346 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos version 393985 summary v 393985 2013-04-03 10:34:43.923364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:43.923406 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos 2013-04-03 10:34:43.923421 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos version 393985 summary v 393985 2013-04-03 10:34:43.923430 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:43.923442 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:34:43.923461 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:43.923473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:34:43.923484 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 logging 2013-04-03 10:34:43.873213 mon.0 10.252.24.25:6789/0 39 : [INF] osd.2 10.252.24.30:6800/1508 boot 2013-04-03 10:34:43.923501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 logging 2013-04-03 10:34:43.873634 mon.0 10.252.24.25:6789/0 40 : [INF] osdmap e601: 5 osds: 5 up, 5 in 2013-04-03 10:34:43.958817 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos 2013-04-03 10:34:43.958834 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos version 393985 summary v 393985 2013-04-03 10:34:43.958850 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:43.958877 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:43.958885 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2224380 from 10.252.24.88:6789/0 2013-04-03 10:34:43.958909 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_query osd_alive(want up_thru 601 have 601) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:43.958917 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_alive want up_thru 601 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:43.958921 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_update osd_alive(want up_thru 601 have 601) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:43.958926 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_alive want up_thru 601 have 601 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:43.958930 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 should_propose 2013-04-03 10:34:43.958948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:43.958953 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2294380 from 10.252.24.88:6789/0 2013-04-03 10:34:43.958974 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_query osd_alive(want up_thru 601 have 601) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:43.958981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_alive want up_thru 601 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:43.958985 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_update osd_alive(want up_thru 601 have 601) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:43.958988 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_alive want up_thru 601 have 601 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:43.958991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 should_propose 2013-04-03 10:34:43.959008 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:43.959013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2294700 from 10.252.24.88:6789/0 2013-04-03 10:34:43.959033 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_query osd_alive(want up_thru 601 have 601) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:43.959040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_alive want up_thru 601 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:43.959043 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_update osd_alive(want up_thru 601 have 601) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:43.959047 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_alive want up_thru 601 have 601 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:43.959050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 should_propose 2013-04-03 10:34:43.992304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:43.992313 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2294a80 from 10.252.24.88:6789/0 2013-04-03 10:34:43.992351 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_query osd_alive(want up_thru 601 have 601) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:43.992358 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_alive want up_thru 601 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:43.992362 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_update osd_alive(want up_thru 601 have 601) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:43.992366 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_alive want up_thru 601 have 601 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:43.992370 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 should_propose 2013-04-03 10:34:44.871381 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 encode_pending v393986 2013-04-03 10:34:45.021251 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos 2013-04-03 10:34:45.021288 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos version 393986 summary v 393985 2013-04-03 10:34:45.021303 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos latest full 393798 2013-04-03 10:34:45.021332 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos applying incremental log 393986 2013-04-03 10:34:42.690351 mon.0 10.252.24.25:6789/0 38 : [INF] pgmap v28979: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 4526B/s wr, 1op/s 2013-04-03 10:34:45.021359 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos applying incremental log 393986 2013-04-03 10:34:43.873213 mon.0 10.252.24.25:6789/0 39 : [INF] osd.2 10.252.24.30:6800/1508 boot 2013-04-03 10:34:45.021379 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos applying incremental log 393986 2013-04-03 10:34:43.873634 mon.0 10.252.24.25:6789/0 40 : [INF] osdmap e601: 5 osds: 5 up, 5 in 2013-04-03 10:34:45.021416 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 check_subs 2013-04-03 10:34:45.021445 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 create_pending v 393987 2013-04-03 10:34:45.021467 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:45.021490 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:45.021553 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos 2013-04-03 10:34:45.021569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos version 393986 summary v 393986 2013-04-03 10:34:45.021586 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:45.021650 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 min_last_epoch_clean 593 2013-04-03 10:34:45.021706 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 min_last_epoch_clean 593 2013-04-03 10:34:45.021757 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 encode_pending e 602 2013-04-03 10:34:45.050839 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:45.050848 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20e3840 from 10.252.24.88:6789/0 2013-04-03 10:34:45.050881 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 19 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:45.063249 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 update_from_paxos applying incremental 602 2013-04-03 10:34:45.063293 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:34:45.064402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 check_osd_map applying osdmap e602 to pg_map 2013-04-03 10:34:45.064425 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 register_new_pgs checking pg pools for osdmap epoch 602, last_pg_scan 341 2013-04-03 10:34:45.064429 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 2013-04-03 10:34:45.064434 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:34:45.064437 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:34:45.064439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 2013-04-03 10:34:45.064442 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 2013-04-03 10:34:45.064445 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 2013-04-03 10:34:45.064448 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 2013-04-03 10:34:45.064451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 2013-04-03 10:34:45.064453 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 2013-04-03 10:34:45.064464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs 2013-04-03 10:34:45.064466 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 check_down_pgs 2013-04-03 10:34:45.064525 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 encode_pending v 28981 2013-04-03 10:34:45.065181 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 send_pg_creates to 0 pgs 2013-04-03 10:34:45.065193 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_to_waiting 602 2013-04-03 10:34:45.065197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 share_map_with_random_osd no up osd on our session map 2013-04-03 10:34:45.065199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 update_logger 2013-04-03 10:34:45.065222 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 create_pending e 603 2013-04-03 10:34:45.065226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 remove_redundant_pg_temp 2013-04-03 10:34:45.065229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 remove_down_pg_temp 2013-04-03 10:34:45.065247 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 _reply_map 601 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:45.065267 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_latest to osd.1 10.252.24.25:6805/3572 start 601 2013-04-03 10:34:45.065272 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_incremental [601..602] to osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:45.065284 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 build_incremental [601..602] 2013-04-03 10:34:45.065329 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 _reply_map 601 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:45.065347 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_latest to osd.2 10.252.24.30:6800/1508 start 601 2013-04-03 10:34:45.065352 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_incremental [601..602] to osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:45.065363 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 build_incremental [601..602] 2013-04-03 10:34:45.065410 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 _reply_map 601 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:45.065429 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_latest to osd.3 10.252.24.88:6801/17538 start 601 2013-04-03 10:34:45.065434 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_incremental [601..602] to osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:45.065445 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 build_incremental [601..602] 2013-04-03 10:34:45.065485 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 _reply_map 601 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:45.065503 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_latest to osd.0 10.252.58.31:6802/2682 start 601 2013-04-03 10:34:45.065508 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_incremental [601..602] to osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:45.065519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 build_incremental [601..602] 2013-04-03 10:34:45.065560 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 update_logger 2013-04-03 10:34:45.065568 7fcdffcdb700 0 log [INF] : osdmap e602: 5 osds: 5 up, 5 in 2013-04-03 10:34:45.065634 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos 2013-04-03 10:34:45.065650 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos version 393986 summary v 393986 2013-04-03 10:34:45.065660 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:45.065672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:34:45.065692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:45.065705 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:34:45.065716 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 logging 2013-04-03 10:34:43.923257 mon.0 10.252.24.25:6789/0 41 : [INF] pgmap v28980: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:45.065737 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 logging 2013-04-03 10:34:45.065569 mon.0 10.252.24.25:6789/0 42 : [INF] osdmap e602: 5 osds: 5 up, 5 in 2013-04-03 10:34:45.113337 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 update_from_paxos applying incremental 28981 2013-04-03 10:34:45.113347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 v28981: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:45.113366 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 send_pg_creates to 0 pgs 2013-04-03 10:34:45.113369 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 update_logger 2013-04-03 10:34:45.113379 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 create_pending v 28982 2013-04-03 10:34:45.113388 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 check_osd_map already seen 602 >= 602 2013-04-03 10:34:45.113390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 update_logger 2013-04-03 10:34:45.113403 7fcdffcdb700 0 log [INF] : pgmap v28981: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:45.113464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos 2013-04-03 10:34:45.113480 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos version 393986 summary v 393986 2013-04-03 10:34:45.113497 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:45.113553 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos 2013-04-03 10:34:45.113567 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos version 393986 summary v 393986 2013-04-03 10:34:45.113575 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:45.113585 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:45.113602 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 nothing new 2013-04-03 10:34:45.151232 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos 2013-04-03 10:34:45.151249 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos version 393986 summary v 393986 2013-04-03 10:34:45.151264 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:45.377079 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21675 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:34:46.062790 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 encode_pending v393987 2013-04-03 10:34:46.150084 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos 2013-04-03 10:34:46.150112 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos version 393987 summary v 393986 2013-04-03 10:34:46.150127 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos latest full 393798 2013-04-03 10:34:46.150145 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos applying incremental log 393987 2013-04-03 10:34:43.923257 mon.0 10.252.24.25:6789/0 41 : [INF] pgmap v28980: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:46.150171 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos applying incremental log 393987 2013-04-03 10:34:45.065569 mon.0 10.252.24.25:6789/0 42 : [INF] osdmap e602: 5 osds: 5 up, 5 in 2013-04-03 10:34:46.150208 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 check_subs 2013-04-03 10:34:46.150237 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 create_pending v 393988 2013-04-03 10:34:46.150260 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:46.150322 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos 2013-04-03 10:34:46.150338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos version 393987 summary v 393987 2013-04-03 10:34:46.150357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:46.175148 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos 2013-04-03 10:34:46.175165 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos version 393987 summary v 393987 2013-04-03 10:34:46.175180 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:47.577077 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140305 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:34:48.324315 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 check_down_pgs 2013-04-03 10:34:48.324345 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 v28981: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:48.324374 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:34:48.324396 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:34:48.324433 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:34:48.324464 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos 2013-04-03 10:34:48.324482 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos version 393987 summary v 393987 2013-04-03 10:34:48.324492 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 log 2013-04-03 10:34:48.324514 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:48.324533 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:34:48.408292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:48.408301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20ad240 from 10.252.24.88:6789/0 2013-04-03 10:34:48.408337 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 preprocess_query pg_stats(0 pgs tid 57 v 602) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:34:48.408345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 prepare_update pg_stats(0 pgs tid 57 v 602) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:34:48.408349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 prepare_pg_stats pg_stats(0 pgs tid 57 v 602) v1 from osd.4 2013-04-03 10:34:48.408355 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 got osd.4 osd_stat(1051 MB used, 56606 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1052 MB used, 56605 MB avail, 59728 MB total, peers []/[])) 2013-04-03 10:34:48.458525 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 encode_pending v 28982 2013-04-03 10:34:48.543385 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 update_from_paxos applying incremental 28982 2013-04-03 10:34:48.543396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 v28982: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5673 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:48.543413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 send_pg_creates to 0 pgs 2013-04-03 10:34:48.543417 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 update_logger 2013-04-03 10:34:48.543426 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 create_pending v 28983 2013-04-03 10:34:48.543436 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 _updated_stats for osd.4 10.252.24.32:6800/13473 2013-04-03 10:34:48.543451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 check_osd_map already seen 602 >= 602 2013-04-03 10:34:48.543455 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 update_logger 2013-04-03 10:34:48.543467 7fcdffcdb700 0 log [INF] : pgmap v28982: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5673 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:48.543530 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos 2013-04-03 10:34:48.543547 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos version 393987 summary v 393987 2013-04-03 10:34:48.543565 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:48.543608 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos 2013-04-03 10:34:48.543623 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos version 393987 summary v 393987 2013-04-03 10:34:48.543632 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:48.543645 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:34:48.543669 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:48.543682 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:34:48.543692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 logging 2013-04-03 10:34:45.113404 mon.0 10.252.24.25:6789/0 43 : [INF] pgmap v28981: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:48.543710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 logging 2013-04-03 10:34:48.543468 mon.0 10.252.24.25:6789/0 44 : [INF] pgmap v28982: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5673 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:48.601555 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos 2013-04-03 10:34:48.601571 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos version 393987 summary v 393987 2013-04-03 10:34:48.601586 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:48.604268 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:48.604275 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20ad900 from 10.252.24.88:6789/0 2013-04-03 10:34:48.604655 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 preprocess_query pg_stats(71 pgs tid 1 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:48.604667 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 prepare_update pg_stats(71 pgs tid 1 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:48.604672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 prepare_pg_stats pg_stats(71 pgs tid 1 v 602) v1 from osd.2 2013-04-03 10:34:48.604677 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[])) 2013-04-03 10:34:48.922831 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:48.922839 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20e4b80 from 10.252.24.88:6789/0 2013-04-03 10:34:48.922872 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 20 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:49.377177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21676 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:34:49.543169 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 encode_pending v393988 2013-04-03 10:34:49.544394 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 encode_pending v 28983 2013-04-03 10:34:49.784199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos 2013-04-03 10:34:49.784226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos version 393988 summary v 393987 2013-04-03 10:34:49.784241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos latest full 393798 2013-04-03 10:34:49.784260 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos applying incremental log 393988 2013-04-03 10:34:45.113404 mon.0 10.252.24.25:6789/0 43 : [INF] pgmap v28981: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:49.784286 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos applying incremental log 393988 2013-04-03 10:34:48.543468 mon.0 10.252.24.25:6789/0 44 : [INF] pgmap v28982: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5673 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:49.784325 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 check_subs 2013-04-03 10:34:49.784354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 create_pending v 393989 2013-04-03 10:34:49.784377 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:49.841367 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:49.841376 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20ad240 from 10.252.24.88:6789/0 2013-04-03 10:34:49.843780 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:49.843787 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:34:49.935566 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 update_from_paxos applying incremental 28983 2013-04-03 10:34:49.935756 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 v28983: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:49.935787 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 send_pg_creates to 0 pgs 2013-04-03 10:34:49.935792 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 update_logger 2013-04-03 10:34:49.935816 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 create_pending v 28984 2013-04-03 10:34:49.935827 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 _updated_stats for osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:49.935877 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 preprocess_query pg_stats(45 pgs tid 9 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:49.935886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 prepare_update pg_stats(45 pgs tid 9 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:49.935891 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 prepare_pg_stats pg_stats(45 pgs tid 9 v 602) v1 from osd.1 2013-04-03 10:34:49.935897 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:34:49.935972 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 preprocess_query pg_stats(73 pgs tid 30435 v 602) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:49.935981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 prepare_update pg_stats(73 pgs tid 30435 v 602) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:49.935985 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 prepare_pg_stats pg_stats(73 pgs tid 30435 v 602) v1 from osd.3 2013-04-03 10:34:49.935989 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 got osd.3 osd_stat(1190 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1192 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:34:49.936076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 check_osd_map already seen 602 >= 602 2013-04-03 10:34:49.936082 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 update_logger 2013-04-03 10:34:49.936093 7fcdffcdb700 0 log [INF] : pgmap v28983: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:49.936161 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos 2013-04-03 10:34:49.936180 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos version 393988 summary v 393988 2013-04-03 10:34:49.936199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:49.936245 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos 2013-04-03 10:34:49.936260 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos version 393988 summary v 393988 2013-04-03 10:34:49.936269 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:49.936284 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:49.936303 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:49.936316 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:49.936326 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 logging 2013-04-03 10:34:49.936094 mon.0 10.252.24.25:6789/0 45 : [INF] pgmap v28983: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:49.967803 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos 2013-04-03 10:34:49.967823 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos version 393988 summary v 393988 2013-04-03 10:34:49.967840 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:49.969869 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:49.969879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:34:49.969920 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 preprocess_query pg_stats(51 pgs tid 406 v 602) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:49.969929 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 prepare_update pg_stats(51 pgs tid 406 v 602) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:49.969933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 prepare_pg_stats pg_stats(51 pgs tid 406 v 602) v1 from osd.0 2013-04-03 10:34:49.969940 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 got osd.0 osd_stat(1138 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:34:50.934363 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 encode_pending v393989 2013-04-03 10:34:50.935452 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 encode_pending v 28984 2013-04-03 10:34:50.967231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos 2013-04-03 10:34:50.967259 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos version 393989 summary v 393988 2013-04-03 10:34:50.967274 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos latest full 393798 2013-04-03 10:34:50.967293 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos applying incremental log 393989 2013-04-03 10:34:49.936094 mon.0 10.252.24.25:6789/0 45 : [INF] pgmap v28983: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:34:50.967339 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 check_subs 2013-04-03 10:34:50.967370 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 create_pending v 393990 2013-04-03 10:34:50.967393 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:51.005965 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 update_from_paxos applying incremental 28984 2013-04-03 10:34:51.006364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 v28984: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 4547B/s wr, 2op/s 2013-04-03 10:34:51.006402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 send_pg_creates to 0 pgs 2013-04-03 10:34:51.006407 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 update_logger 2013-04-03 10:34:51.006431 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 create_pending v 28985 2013-04-03 10:34:51.006443 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:51.006468 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:34:51.006511 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:34:51.007660 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 check_osd_map already seen 602 >= 602 2013-04-03 10:34:51.007672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 update_logger 2013-04-03 10:34:51.007696 7fcdffcdb700 0 log [INF] : pgmap v28984: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 4547B/s wr, 2op/s 2013-04-03 10:34:51.007963 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos 2013-04-03 10:34:51.007983 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos version 393989 summary v 393989 2013-04-03 10:34:51.008002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:51.008046 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos 2013-04-03 10:34:51.008061 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos version 393989 summary v 393989 2013-04-03 10:34:51.008070 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:51.008082 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:51.008098 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:51.008109 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:51.008120 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 logging 2013-04-03 10:34:51.007704 mon.0 10.252.24.25:6789/0 46 : [INF] pgmap v28984: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 4547B/s wr, 2op/s 2013-04-03 10:34:51.037578 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos 2013-04-03 10:34:51.037595 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos version 393989 summary v 393989 2013-04-03 10:34:51.037787 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:51.577203 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140306 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:34:52.004296 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 encode_pending v393990 2013-04-03 10:34:52.112696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos 2013-04-03 10:34:52.112736 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos version 393990 summary v 393989 2013-04-03 10:34:52.112751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos latest full 393798 2013-04-03 10:34:52.112770 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos applying incremental log 393990 2013-04-03 10:34:51.007704 mon.0 10.252.24.25:6789/0 46 : [INF] pgmap v28984: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 4547B/s wr, 2op/s 2013-04-03 10:34:52.112818 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 check_subs 2013-04-03 10:34:52.112848 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 create_pending v 393991 2013-04-03 10:34:52.112872 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:52.112934 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos 2013-04-03 10:34:52.112950 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos version 393990 summary v 393990 2013-04-03 10:34:52.112969 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:52.123501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos 2013-04-03 10:34:52.123517 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos version 393990 summary v 393990 2013-04-03 10:34:52.123534 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:52.922942 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:52.922952 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273a580 from 10.252.24.88:6789/0 2013-04-03 10:34:52.922989 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 21 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:53.324671 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 check_down_pgs 2013-04-03 10:34:53.324701 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 v28984: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 4547B/s wr, 2op/s 2013-04-03 10:34:53.324731 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:34:53.324755 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:34:53.324793 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:34:53.324825 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos 2013-04-03 10:34:53.324843 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos version 393990 summary v 393990 2013-04-03 10:34:53.324853 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 log 2013-04-03 10:34:53.324873 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:53.324891 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:34:53.377341 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21677 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:34:54.557349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:54.557360 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:34:54.557411 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 preprocess_query pg_stats(1 pgs tid 10 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:54.557419 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 prepare_update pg_stats(1 pgs tid 10 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:54.557423 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 prepare_pg_stats pg_stats(1 pgs tid 10 v 602) v1 from osd.1 2013-04-03 10:34:54.557432 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:34:54.607612 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 encode_pending v 28985 2013-04-03 10:34:54.648772 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 update_from_paxos applying incremental 28985 2013-04-03 10:34:54.648788 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 v28985: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 2629B/s wr, 1op/s 2013-04-03 10:34:54.648808 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 send_pg_creates to 0 pgs 2013-04-03 10:34:54.648813 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 update_logger 2013-04-03 10:34:54.648823 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 create_pending v 28986 2013-04-03 10:34:54.648832 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:34:54.648849 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 check_osd_map already seen 602 >= 602 2013-04-03 10:34:54.648852 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 update_logger 2013-04-03 10:34:54.648863 7fcdffcdb700 0 log [INF] : pgmap v28985: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 2629B/s wr, 1op/s 2013-04-03 10:34:54.648921 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos 2013-04-03 10:34:54.648942 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos version 393990 summary v 393990 2013-04-03 10:34:54.648961 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:54.649003 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos 2013-04-03 10:34:54.649018 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos version 393990 summary v 393990 2013-04-03 10:34:54.649028 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:54.649040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:54.649059 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:54.649072 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:54.649084 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 logging 2013-04-03 10:34:54.648864 mon.0 10.252.24.25:6789/0 47 : [INF] pgmap v28985: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 2629B/s wr, 1op/s 2013-04-03 10:34:54.676970 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos 2013-04-03 10:34:54.676987 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos version 393990 summary v 393990 2013-04-03 10:34:54.677002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:55.577339 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140307 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:34:55.642731 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 encode_pending v393991 2013-04-03 10:34:55.670055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos 2013-04-03 10:34:55.670082 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos version 393991 summary v 393990 2013-04-03 10:34:55.670097 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos latest full 393798 2013-04-03 10:34:55.670115 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos applying incremental log 393991 2013-04-03 10:34:54.648864 mon.0 10.252.24.25:6789/0 47 : [INF] pgmap v28985: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 2629B/s wr, 1op/s 2013-04-03 10:34:55.670170 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 check_subs 2013-04-03 10:34:55.670202 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 create_pending v 393992 2013-04-03 10:34:55.670225 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:55.670286 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos 2013-04-03 10:34:55.670302 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos version 393991 summary v 393991 2013-04-03 10:34:55.670321 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:55.682148 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos 2013-04-03 10:34:55.682164 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos version 393991 summary v 393991 2013-04-03 10:34:55.682181 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:56.923157 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:56.923168 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273a2c0 from 10.252.24.88:6789/0 2013-04-03 10:34:56.923221 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 22 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:34:57.377360 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21678 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:34:58.325040 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 check_down_pgs 2013-04-03 10:34:58.325069 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 v28985: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 2629B/s wr, 1op/s 2013-04-03 10:34:58.325097 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:34:58.325121 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:34:58.325159 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:34:58.325190 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos 2013-04-03 10:34:58.325212 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos version 393991 summary v 393991 2013-04-03 10:34:58.325222 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 log 2013-04-03 10:34:58.325239 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:58.325258 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:34:58.477494 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 2013-04-03 10:34:58.477504 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:34:58.477539 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 preprocess_query pg_stats(1 pgs tid 2 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:58.477547 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 prepare_update pg_stats(1 pgs tid 2 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:58.477551 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 prepare_pg_stats pg_stats(1 pgs tid 2 v 602) v1 from osd.2 2013-04-03 10:34:58.477557 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:34:58.527734 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 encode_pending v 28986 2013-04-03 10:34:58.714037 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 update_from_paxos applying incremental 28986 2013-04-03 10:34:58.714055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 v28986: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s 2013-04-03 10:34:58.714073 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 send_pg_creates to 0 pgs 2013-04-03 10:34:58.714078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 update_logger 2013-04-03 10:34:58.714088 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 create_pending v 28987 2013-04-03 10:34:58.714098 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 _updated_stats for osd.2 10.252.24.30:6800/1508 2013-04-03 10:34:58.714113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 check_osd_map already seen 602 >= 602 2013-04-03 10:34:58.714118 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 update_logger 2013-04-03 10:34:58.714129 7fcdffcdb700 0 log [INF] : pgmap v28986: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s 2013-04-03 10:34:58.714187 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos 2013-04-03 10:34:58.714205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos version 393991 summary v 393991 2013-04-03 10:34:58.714223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:58.714264 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos 2013-04-03 10:34:58.714280 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos version 393991 summary v 393991 2013-04-03 10:34:58.714291 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:58.714303 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:34:58.714323 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:34:58.714336 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:34:58.714347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 logging 2013-04-03 10:34:58.714131 mon.0 10.252.24.25:6789/0 48 : [INF] pgmap v28986: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s 2013-04-03 10:34:58.727296 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos 2013-04-03 10:34:58.727313 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos version 393991 summary v 393991 2013-04-03 10:34:58.727328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:59.577474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140308 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:34:59.713797 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 encode_pending v393992 2013-04-03 10:34:59.752713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos 2013-04-03 10:34:59.752740 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393991 2013-04-03 10:34:59.752754 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos latest full 393798 2013-04-03 10:34:59.752773 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos applying incremental log 393992 2013-04-03 10:34:58.714131 mon.0 10.252.24.25:6789/0 48 : [INF] pgmap v28986: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s 2013-04-03 10:34:59.752821 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 check_subs 2013-04-03 10:34:59.752851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 create_pending v 393993 2013-04-03 10:34:59.752874 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:34:59.752935 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos 2013-04-03 10:34:59.752951 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393992 2013-04-03 10:34:59.752969 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:34:59.764775 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos 2013-04-03 10:34:59.764792 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393992 2013-04-03 10:34:59.764809 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:00.153219 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x26dc000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:32963/0) 2013-04-03 10:35:00.923247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:00.923258 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20e3dc0 from 10.252.24.88:6789/0 2013-04-03 10:35:00.923310 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 23 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:35:01.377487 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21679 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:35:02.753543 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos 2013-04-03 10:35:02.753572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393992 2013-04-03 10:35:02.753590 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:03.325405 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 check_down_pgs 2013-04-03 10:35:03.325430 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 v28986: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s 2013-04-03 10:35:03.325457 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:35:03.325481 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:35:03.325519 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:35:03.325550 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos 2013-04-03 10:35:03.325571 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393992 2013-04-03 10:35:03.325581 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 log 2013-04-03 10:35:03.325598 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:03.325617 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:35:03.478123 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:03.478133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:35:03.478169 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 preprocess_query pg_stats(1 pgs tid 3 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:35:03.478177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 prepare_update pg_stats(1 pgs tid 3 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:35:03.478181 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 prepare_pg_stats pg_stats(1 pgs tid 3 v 602) v1 from osd.2 2013-04-03 10:35:03.478187 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:35:03.528367 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 encode_pending v 28987 2013-04-03 10:35:03.577569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140309 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:35:03.712428 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 update_from_paxos applying incremental 28987 2013-04-03 10:35:03.712444 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28987 v28987: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3214B/s wr, 0op/s 2013-04-03 10:35:03.712463 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28987 send_pg_creates to 0 pgs 2013-04-03 10:35:03.712467 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28987 update_logger 2013-04-03 10:35:03.712476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28987 create_pending v 28988 2013-04-03 10:35:03.712486 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28987 _updated_stats for osd.2 10.252.24.30:6800/1508 2013-04-03 10:35:03.712502 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28987 check_osd_map already seen 602 >= 602 2013-04-03 10:35:03.712506 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28987 update_logger 2013-04-03 10:35:03.712517 7fcdffcdb700 0 log [INF] : pgmap v28987: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3214B/s wr, 0op/s 2013-04-03 10:35:03.712575 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos 2013-04-03 10:35:03.712593 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393992 2013-04-03 10:35:03.712610 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:03.712652 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos 2013-04-03 10:35:03.712667 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393992 2013-04-03 10:35:03.712676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:03.712687 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:35:03.712706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:03.712719 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:35:03.712729 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 logging 2013-04-03 10:35:03.712518 mon.0 10.252.24.25:6789/0 49 : [INF] pgmap v28987: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3214B/s wr, 0op/s 2013-04-03 10:35:03.738108 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos 2013-04-03 10:35:03.738125 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393992 2013-04-03 10:35:03.738141 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:04.712233 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 encode_pending v393993 2013-04-03 10:35:04.751978 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos 2013-04-03 10:35:04.752207 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos version 393993 summary v 393992 2013-04-03 10:35:04.752222 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos latest full 393798 2013-04-03 10:35:04.752243 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos applying incremental log 393993 2013-04-03 10:35:03.712518 mon.0 10.252.24.25:6789/0 49 : [INF] pgmap v28987: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3214B/s wr, 0op/s 2013-04-03 10:35:04.752288 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 check_subs 2013-04-03 10:35:04.752317 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 create_pending v 393994 2013-04-03 10:35:04.752340 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:35:04.752402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos 2013-04-03 10:35:04.752418 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos version 393993 summary v 393993 2013-04-03 10:35:04.752436 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:04.765275 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos 2013-04-03 10:35:04.765292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos version 393993 summary v 393993 2013-04-03 10:35:04.765308 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:04.923353 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:04.923364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20e48c0 from 10.252.24.88:6789/0 2013-04-03 10:35:04.923396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 24 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:35:05.377612 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21680 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:35:07.577713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140310 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:35:07.752892 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos 2013-04-03 10:35:07.752911 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos version 393993 summary v 393993 2013-04-03 10:35:07.752928 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:08.157374 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) 2013-04-03 10:35:08.157417 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader 2013-04-03 10:35:08.325769 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28987 check_down_pgs 2013-04-03 10:35:08.325796 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28987 v28987: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3214B/s wr, 0op/s 2013-04-03 10:35:08.325823 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:35:08.325844 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:35:08.325883 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:35:08.325913 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos 2013-04-03 10:35:08.325930 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos version 393993 summary v 393993 2013-04-03 10:35:08.325939 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393993 log 2013-04-03 10:35:08.325955 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:08.325973 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:35:08.923506 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:08.923515 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4340 from 10.252.24.88:6789/0 2013-04-03 10:35:08.923552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 25 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:35:09.377723 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21681 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:35:09.558952 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:09.558961 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:35:09.558994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28987 preprocess_query pg_stats(1 pgs tid 11 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:35:09.559002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28987 prepare_update pg_stats(1 pgs tid 11 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:35:09.559005 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28987 prepare_pg_stats pg_stats(1 pgs tid 11 v 602) v1 from osd.1 2013-04-03 10:35:09.559011 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28987 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:35:09.609198 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28987 encode_pending v 28988 2013-04-03 10:35:09.642465 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28987 update_from_paxos applying incremental 28988 2013-04-03 10:35:09.642480 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 v28988: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 1848B/s wr, 0op/s 2013-04-03 10:35:09.642499 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 send_pg_creates to 0 pgs 2013-04-03 10:35:09.642502 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 update_logger 2013-04-03 10:35:09.642511 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 create_pending v 28989 2013-04-03 10:35:09.642519 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28988 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:35:09.642537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 check_osd_map already seen 602 >= 602 2013-04-03 10:35:09.642540 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 update_logger 2013-04-03 10:35:09.642550 7fcdffcdb700 0 log [INF] : pgmap v28988: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 1848B/s wr, 0op/s 2013-04-03 10:35:09.642608 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos 2013-04-03 10:35:09.642625 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos version 393993 summary v 393993 2013-04-03 10:35:09.642644 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:09.642686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos 2013-04-03 10:35:09.642701 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos version 393993 summary v 393993 2013-04-03 10:35:09.642708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:09.642720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:35:09.642738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:09.642749 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:35:09.642759 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 logging 2013-04-03 10:35:09.642551 mon.0 10.252.24.25:6789/0 50 : [INF] pgmap v28988: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 1848B/s wr, 0op/s 2013-04-03 10:35:09.654382 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos 2013-04-03 10:35:09.654398 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos version 393993 summary v 393993 2013-04-03 10:35:09.654906 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:09.726945 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:09.726952 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:35:09.726981 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 preprocess_query pg_stats(1 pgs tid 30436 v 602) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:35:09.726988 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 prepare_update pg_stats(1 pgs tid 30436 v 602) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:35:09.726991 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 prepare_pg_stats pg_stats(1 pgs tid 30436 v 602) v1 from osd.3 2013-04-03 10:35:09.726996 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 got osd.3 osd_stat(1191 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1190 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:35:10.642182 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393993 encode_pending v393994 2013-04-03 10:35:10.643446 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28988 encode_full pgmap v 28988 2013-04-03 10:35:10.644063 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28988 encode_pending v 28989 2013-04-03 10:35:10.722552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos 2013-04-03 10:35:10.722578 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos version 393994 summary v 393993 2013-04-03 10:35:10.722592 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos latest full 393798 2013-04-03 10:35:10.722610 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393994 update_from_paxos applying incremental log 393994 2013-04-03 10:35:09.642551 mon.0 10.252.24.25:6789/0 50 : [INF] pgmap v28988: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 1848B/s wr, 0op/s 2013-04-03 10:35:10.722653 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 check_subs 2013-04-03 10:35:10.722683 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 create_pending v 393995 2013-04-03 10:35:10.722704 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393994 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:35:10.776000 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28988 update_from_paxos applying incremental 28989 2013-04-03 10:35:10.776018 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 v28989: 240 pgs: 240 active+clean; 252 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s 2013-04-03 10:35:10.776659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 send_pg_creates to 0 pgs 2013-04-03 10:35:10.776664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 update_logger 2013-04-03 10:35:10.776673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 create_pending v 28990 2013-04-03 10:35:10.776682 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28989 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:35:10.776698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 check_osd_map already seen 602 >= 602 2013-04-03 10:35:10.776702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 update_logger 2013-04-03 10:35:10.776712 7fcdffcdb700 0 log [INF] : pgmap v28989: 240 pgs: 240 active+clean; 252 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s 2013-04-03 10:35:10.776769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos 2013-04-03 10:35:10.776785 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos version 393994 summary v 393994 2013-04-03 10:35:10.776803 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:10.776844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos 2013-04-03 10:35:10.776858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos version 393994 summary v 393994 2013-04-03 10:35:10.776867 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:10.776877 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:35:10.776894 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:10.776906 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:35:10.776915 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 logging 2013-04-03 10:35:10.776713 mon.0 10.252.24.25:6789/0 51 : [INF] pgmap v28989: 240 pgs: 240 active+clean; 252 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s 2013-04-03 10:35:10.833759 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos 2013-04-03 10:35:10.833777 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos version 393994 summary v 393994 2013-04-03 10:35:10.833792 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:11.577831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140311 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:35:11.774443 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393994 encode_pending v393995 2013-04-03 10:35:11.803977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos 2013-04-03 10:35:11.804003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393994 2013-04-03 10:35:11.804017 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos latest full 393798 2013-04-03 10:35:11.804034 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393995 update_from_paxos applying incremental log 393995 2013-04-03 10:35:10.776713 mon.0 10.252.24.25:6789/0 51 : [INF] pgmap v28989: 240 pgs: 240 active+clean; 252 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s 2013-04-03 10:35:11.804077 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 check_subs 2013-04-03 10:35:11.804106 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 create_pending v 393996 2013-04-03 10:35:11.804127 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393995 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:35:11.804188 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos 2013-04-03 10:35:11.804203 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 2013-04-03 10:35:11.804220 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:11.815999 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos 2013-04-03 10:35:11.816015 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 2013-04-03 10:35:11.816030 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:12.923673 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:12.923683 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4b80 from 10.252.24.88:6789/0 2013-04-03 10:35:12.923734 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 26 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:35:13.326117 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28989 check_down_pgs 2013-04-03 10:35:13.326151 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28989 v28989: 240 pgs: 240 active+clean; 252 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s 2013-04-03 10:35:13.326177 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:35:13.326198 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:35:13.326235 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:35:13.326264 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos 2013-04-03 10:35:13.326282 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 2013-04-03 10:35:13.326291 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393995 log 2013-04-03 10:35:13.326307 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:13.326325 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:35:13.377780 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21682 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:35:14.804858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos 2013-04-03 10:35:14.804877 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 2013-04-03 10:35:14.804894 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:15.577948 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140312 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:35:16.923779 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:16.923789 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2146b80 from 10.252.24.88:6789/0 2013-04-03 10:35:16.923839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 27 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:35:17.377960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21683 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:35:17.805059 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos 2013-04-03 10:35:17.805078 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 2013-04-03 10:35:17.805096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:18.326472 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28989 v28989: 240 pgs: 240 active+clean; 252 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s 2013-04-03 10:35:18.326511 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:35:18.326536 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:35:18.326581 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:35:18.326614 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos 2013-04-03 10:35:18.326632 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 2013-04-03 10:35:18.326641 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393995 log 2013-04-03 10:35:18.326657 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:18.326674 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:35:18.480188 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:18.480198 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:35:18.480247 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 preprocess_query pg_stats(1 pgs tid 4 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:35:18.480273 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 prepare_update pg_stats(1 pgs tid 4 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:35:18.480278 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 prepare_pg_stats pg_stats(1 pgs tid 4 v 602) v1 from osd.2 2013-04-03 10:35:18.480285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 got osd.2 osd_stat(1173 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:35:18.530468 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28989 encode_pending v 28990 2013-04-03 10:35:18.559014 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28989 update_from_paxos applying incremental 28990 2013-04-03 10:35:18.559028 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 v28990: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 81380B/s wr, 0op/s 2013-04-03 10:35:18.559045 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 send_pg_creates to 0 pgs 2013-04-03 10:35:18.559047 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 update_logger 2013-04-03 10:35:18.559057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 create_pending v 28991 2013-04-03 10:35:18.559065 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28990 _updated_stats for osd.2 10.252.24.30:6800/1508 2013-04-03 10:35:18.559082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 check_osd_map already seen 602 >= 602 2013-04-03 10:35:18.559086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 update_logger 2013-04-03 10:35:18.559096 7fcdffcdb700 0 log [INF] : pgmap v28990: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 81380B/s wr, 0op/s 2013-04-03 10:35:18.559156 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos 2013-04-03 10:35:18.559174 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 2013-04-03 10:35:18.559192 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:18.559234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos 2013-04-03 10:35:18.559249 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 2013-04-03 10:35:18.559257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:18.559267 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:35:18.559287 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:18.559299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:35:18.559308 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 logging 2013-04-03 10:35:18.559097 mon.0 10.252.24.25:6789/0 52 : [INF] pgmap v28990: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 81380B/s wr, 0op/s 2013-04-03 10:35:18.572320 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos 2013-04-03 10:35:18.572463 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 2013-04-03 10:35:18.572484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:19.559179 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393995 encode_pending v393996 2013-04-03 10:35:19.578038 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140313 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:35:19.589405 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos 2013-04-03 10:35:19.589431 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393995 2013-04-03 10:35:19.589447 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos latest full 393798 2013-04-03 10:35:19.589465 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393996 update_from_paxos applying incremental log 393996 2013-04-03 10:35:18.559097 mon.0 10.252.24.25:6789/0 52 : [INF] pgmap v28990: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 81380B/s wr, 0op/s 2013-04-03 10:35:19.589508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 check_subs 2013-04-03 10:35:19.589538 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 create_pending v 393997 2013-04-03 10:35:19.589559 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393996 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:35:19.589620 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos 2013-04-03 10:35:19.589635 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393996 2013-04-03 10:35:19.589652 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:19.601438 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos 2013-04-03 10:35:19.601455 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393996 2013-04-03 10:35:19.601470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:20.923869 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:20.923878 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2146080 from 10.252.24.88:6789/0 2013-04-03 10:35:20.923929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 28 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:35:21.378066 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21684 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:35:22.590312 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos 2013-04-03 10:35:22.590330 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393996 2013-04-03 10:35:22.590347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:23.326815 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28990 check_down_pgs 2013-04-03 10:35:23.326847 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28990 v28990: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 81380B/s wr, 0op/s 2013-04-03 10:35:23.326873 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:35:23.326897 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:35:23.326933 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:35:23.326971 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos 2013-04-03 10:35:23.326989 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393996 2013-04-03 10:35:23.326998 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393996 log 2013-04-03 10:35:23.327014 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:23.327033 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:35:23.578196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140314 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:35:24.560810 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:24.560823 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:35:24.560874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 preprocess_query pg_stats(1 pgs tid 12 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:35:24.560897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 prepare_update pg_stats(1 pgs tid 12 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:35:24.560903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 prepare_pg_stats pg_stats(1 pgs tid 12 v 602) v1 from osd.1 2013-04-03 10:35:24.560909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:35:24.611093 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28990 encode_pending v 28991 2013-04-03 10:35:24.670036 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28990 update_from_paxos applying incremental 28991 2013-04-03 10:35:24.670051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 v28991: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 52200B/s wr, 0op/s 2013-04-03 10:35:24.670071 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 send_pg_creates to 0 pgs 2013-04-03 10:35:24.670075 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 update_logger 2013-04-03 10:35:24.670085 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 create_pending v 28992 2013-04-03 10:35:24.670093 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28991 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:35:24.670109 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 check_osd_map already seen 602 >= 602 2013-04-03 10:35:24.670112 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 update_logger 2013-04-03 10:35:24.670122 7fcdffcdb700 0 log [INF] : pgmap v28991: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 52200B/s wr, 0op/s 2013-04-03 10:35:24.670186 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos 2013-04-03 10:35:24.670204 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393996 2013-04-03 10:35:24.670222 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:24.670265 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos 2013-04-03 10:35:24.670279 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393996 2013-04-03 10:35:24.670288 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:24.670299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:35:24.670321 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:24.670334 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:35:24.670345 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 logging 2013-04-03 10:35:24.670123 mon.0 10.252.24.25:6789/0 53 : [INF] pgmap v28991: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 52200B/s wr, 0op/s 2013-04-03 10:35:24.685041 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos 2013-04-03 10:35:24.685056 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393996 2013-04-03 10:35:24.685071 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:24.924008 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:24.924016 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae340 from 10.252.24.88:6789/0 2013-04-03 10:35:24.924047 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 29 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:35:25.378174 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21685 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:35:25.669815 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393996 encode_pending v393997 2013-04-03 10:35:25.702111 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos 2013-04-03 10:35:25.702138 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393996 2013-04-03 10:35:25.702152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos latest full 393798 2013-04-03 10:35:25.702170 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393997 update_from_paxos applying incremental log 393997 2013-04-03 10:35:24.670123 mon.0 10.252.24.25:6789/0 53 : [INF] pgmap v28991: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 52200B/s wr, 0op/s 2013-04-03 10:35:25.702214 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 check_subs 2013-04-03 10:35:25.702244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 create_pending v 393998 2013-04-03 10:35:25.702266 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393997 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:35:25.702328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos 2013-04-03 10:35:25.702347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 2013-04-03 10:35:25.702364 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:25.714158 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos 2013-04-03 10:35:25.714174 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 2013-04-03 10:35:25.714188 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:27.578318 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140315 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:35:28.327173 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28991 check_down_pgs 2013-04-03 10:35:28.327204 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28991 v28991: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 52200B/s wr, 0op/s 2013-04-03 10:35:28.327230 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:35:28.327254 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:35:28.327291 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:35:28.327325 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos 2013-04-03 10:35:28.327346 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 2013-04-03 10:35:28.327356 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393997 log 2013-04-03 10:35:28.327372 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:28.327390 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:35:28.703005 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos 2013-04-03 10:35:28.703023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 2013-04-03 10:35:28.703040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:28.924098 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:28.924108 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae600 from 10.252.24.88:6789/0 2013-04-03 10:35:28.924142 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 30 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:35:29.378257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21686 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:35:31.578479 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140316 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:35:31.703033 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos 2013-04-03 10:35:31.703051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 2013-04-03 10:35:31.703068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:32.924302 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:32.924311 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x295eb80 from 10.252.24.88:6789/0 2013-04-03 10:35:32.924362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 31 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:35:33.327529 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28991 v28991: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 52200B/s wr, 0op/s 2013-04-03 10:35:33.327570 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:35:33.327592 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:35:33.327638 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:35:33.327669 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos 2013-04-03 10:35:33.327686 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 2013-04-03 10:35:33.327695 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393997 log 2013-04-03 10:35:33.327712 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:33.327730 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:35:33.378329 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21687 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:35:33.482237 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:33.482245 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:35:33.482275 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 preprocess_query pg_stats(1 pgs tid 5 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:35:33.482296 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 prepare_update pg_stats(1 pgs tid 5 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:35:33.482301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 prepare_pg_stats pg_stats(1 pgs tid 5 v 602) v1 from osd.2 2013-04-03 10:35:33.482307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1173 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:35:33.532491 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28991 encode_pending v 28992 2013-04-03 10:35:33.580201 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28991 update_from_paxos applying incremental 28992 2013-04-03 10:35:33.580221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28992 v28992: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:35:33.580239 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28992 send_pg_creates to 0 pgs 2013-04-03 10:35:33.580243 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28992 update_logger 2013-04-03 10:35:33.580254 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28992 create_pending v 28993 2013-04-03 10:35:33.580263 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28992 _updated_stats for osd.2 10.252.24.30:6800/1508 2013-04-03 10:35:33.580279 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28992 check_osd_map already seen 602 >= 602 2013-04-03 10:35:33.580285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28992 update_logger 2013-04-03 10:35:33.580295 7fcdffcdb700 0 log [INF] : pgmap v28992: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:35:33.580356 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos 2013-04-03 10:35:33.580374 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 2013-04-03 10:35:33.580392 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:33.580434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos 2013-04-03 10:35:33.580449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 2013-04-03 10:35:33.580458 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:33.580469 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:35:33.580489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:33.580501 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:35:33.580510 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 logging 2013-04-03 10:35:33.580296 mon.0 10.252.24.25:6789/0 54 : [INF] pgmap v28992: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:35:33.605776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos 2013-04-03 10:35:33.605794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 2013-04-03 10:35:33.605808 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:34.579923 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393997 encode_pending v393998 2013-04-03 10:35:34.609353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos 2013-04-03 10:35:34.609379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos version 393998 summary v 393997 2013-04-03 10:35:34.609394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos latest full 393798 2013-04-03 10:35:34.609412 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393998 update_from_paxos applying incremental log 393998 2013-04-03 10:35:33.580296 mon.0 10.252.24.25:6789/0 54 : [INF] pgmap v28992: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:35:34.609462 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 check_subs 2013-04-03 10:35:34.609493 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 create_pending v 393999 2013-04-03 10:35:34.609514 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393998 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:35:34.609575 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos 2013-04-03 10:35:34.609590 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos version 393998 summary v 393998 2013-04-03 10:35:34.609608 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:34.621412 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos 2013-04-03 10:35:34.621428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos version 393998 summary v 393998 2013-04-03 10:35:34.621442 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:35.578588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140317 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:35:36.924423 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:36.924432 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x295e600 from 10.252.24.88:6789/0 2013-04-03 10:35:36.924483 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 32 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:35:37.378492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21688 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:35:37.610375 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos 2013-04-03 10:35:37.610394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos version 393998 summary v 393998 2013-04-03 10:35:37.610411 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:38.327879 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28992 check_down_pgs 2013-04-03 10:35:38.327900 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28992 v28992: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:35:38.327925 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:35:38.327949 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:35:38.327985 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:35:38.328015 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos 2013-04-03 10:35:38.328033 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos version 393998 summary v 393998 2013-04-03 10:35:38.328042 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393998 log 2013-04-03 10:35:38.328059 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:35:38.328076 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:35:38.627402 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:35:38.627409 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:35:38.627420 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 2013-04-03 10:35:39.562584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:39.562594 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:35:39.562647 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28992 preprocess_query pg_stats(1 pgs tid 13 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:35:39.562671 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28992 prepare_update pg_stats(1 pgs tid 13 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:35:39.562678 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28992 prepare_pg_stats pg_stats(1 pgs tid 13 v 602) v1 from osd.1 2013-04-03 10:35:39.562684 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28992 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:35:39.578866 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140318 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:35:39.612893 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28992 encode_pending v 28993 2013-04-03 10:35:39.693952 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28992 update_from_paxos applying incremental 28993 2013-04-03 10:35:39.693971 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 v28993: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:35:39.693989 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 send_pg_creates to 0 pgs 2013-04-03 10:35:39.693992 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 update_logger 2013-04-03 10:35:39.694002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 create_pending v 28994 2013-04-03 10:35:39.694011 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:35:39.694028 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 check_osd_map already seen 602 >= 602 2013-04-03 10:35:39.694032 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 update_logger 2013-04-03 10:35:39.694042 7fcdffcdb700 0 log [INF] : pgmap v28993: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:35:39.694109 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 update_from_paxos 2013-04-03 10:35:39.694128 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 update_from_paxos version 393998 summary v 393998 2013-04-03 10:35:39.694147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:39.694190 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 update_from_paxos 2013-04-03 10:35:39.694205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 update_from_paxos version 393998 summary v 393998 2013-04-03 10:35:39.694215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:39.694227 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:35:39.694247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:39.694259 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:35:39.694270 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 logging 2013-04-03 10:35:39.694043 mon.0 10.252.24.25:6789/0 55 : [INF] pgmap v28993: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:35:39.707400 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 update_from_paxos 2013-04-03 10:35:39.707415 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 update_from_paxos version 393998 summary v 393998 2013-04-03 10:35:39.707431 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:39.729982 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:39.729990 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:35:39.730017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 preprocess_query pg_stats(1 pgs tid 30437 v 602) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:35:39.730039 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 prepare_update pg_stats(1 pgs tid 30437 v 602) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:35:39.730046 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 prepare_pg_stats pg_stats(1 pgs tid 30437 v 602) v1 from osd.3 2013-04-03 10:35:39.730051 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1191 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:35:40.693773 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 encode_pending v393999 2013-04-03 10:35:40.694988 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 encode_pending v 28994 2013-04-03 10:35:40.849488 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos 2013-04-03 10:35:40.849526 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos version 393999 summary v 393998 2013-04-03 10:35:40.849541 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos latest full 393798 2013-04-03 10:35:40.849559 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos applying incremental log 393999 2013-04-03 10:35:39.694043 mon.0 10.252.24.25:6789/0 55 : [INF] pgmap v28993: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:35:40.849608 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 check_subs 2013-04-03 10:35:40.849639 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 create_pending v 394000 2013-04-03 10:35:40.849661 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:35:40.876461 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 update_from_paxos applying incremental 28994 2013-04-03 10:35:40.876481 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 v28994: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 714B/s wr, 0op/s 2013-04-03 10:35:40.876501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 send_pg_creates to 0 pgs 2013-04-03 10:35:40.876506 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 update_logger 2013-04-03 10:35:40.876516 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 create_pending v 28995 2013-04-03 10:35:40.876525 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:35:40.876542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 check_osd_map already seen 602 >= 602 2013-04-03 10:35:40.876546 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 update_logger 2013-04-03 10:35:40.876557 7fcdffcdb700 0 log [INF] : pgmap v28994: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 714B/s wr, 0op/s 2013-04-03 10:35:40.876613 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos 2013-04-03 10:35:40.876632 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos version 393999 summary v 393999 2013-04-03 10:35:40.876650 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:40.876692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos 2013-04-03 10:35:40.876707 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos version 393999 summary v 393999 2013-04-03 10:35:40.876717 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:40.876729 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:35:40.876746 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:40.876758 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:35:40.876768 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 logging 2013-04-03 10:35:40.876558 mon.0 10.252.24.25:6789/0 56 : [INF] pgmap v28994: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 714B/s wr, 0op/s 2013-04-03 10:35:40.887373 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos 2013-04-03 10:35:40.887390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos version 393999 summary v 393999 2013-04-03 10:35:40.887406 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:40.924475 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:40.924483 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x295db00 from 10.252.24.88:6789/0 2013-04-03 10:35:40.924513 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 33 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:35:41.378619 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21689 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:35:41.875125 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 encode_pending v394000 2013-04-03 10:35:41.914056 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos 2013-04-03 10:35:41.914083 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 393999 2013-04-03 10:35:41.914098 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos latest full 393798 2013-04-03 10:35:41.914117 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos applying incremental log 394000 2013-04-03 10:35:40.876558 mon.0 10.252.24.25:6789/0 56 : [INF] pgmap v28994: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 714B/s wr, 0op/s 2013-04-03 10:35:41.914166 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 check_subs 2013-04-03 10:35:41.914197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 create_pending v 394001 2013-04-03 10:35:41.914220 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:35:41.914283 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos 2013-04-03 10:35:41.914299 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 2013-04-03 10:35:41.914317 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:41.939612 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos 2013-04-03 10:35:41.939629 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 2013-04-03 10:35:41.939646 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:43.328218 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 check_down_pgs 2013-04-03 10:35:43.328250 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 v28994: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 714B/s wr, 0op/s 2013-04-03 10:35:43.328277 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:35:43.328300 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:35:43.328338 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:35:43.328369 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos 2013-04-03 10:35:43.328388 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 2013-04-03 10:35:43.328401 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 log 2013-04-03 10:35:43.328418 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:43.328437 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:35:43.578781 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140319 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:35:44.914940 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos 2013-04-03 10:35:44.914961 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 2013-04-03 10:35:44.914978 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:44.924533 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:44.924542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x295d840 from 10.252.24.88:6789/0 2013-04-03 10:35:44.924571 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 34 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:35:45.378780 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21690 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:35:47.578931 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140320 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:35:47.915015 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos 2013-04-03 10:35:47.915036 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 2013-04-03 10:35:47.915053 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:48.328583 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 v28994: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 714B/s wr, 0op/s 2013-04-03 10:35:48.328620 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:35:48.328643 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:35:48.328687 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:35:48.328718 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos 2013-04-03 10:35:48.328737 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 2013-04-03 10:35:48.328747 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 log 2013-04-03 10:35:48.328768 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:48.328787 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:35:48.484358 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:48.484368 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:35:48.484400 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 preprocess_query pg_stats(1 pgs tid 6 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:35:48.484421 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 prepare_update pg_stats(1 pgs tid 6 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:35:48.484428 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 prepare_pg_stats pg_stats(1 pgs tid 6 v 602) v1 from osd.2 2013-04-03 10:35:48.484434 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:35:48.534614 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 encode_pending v 28995 2013-04-03 10:35:48.572244 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 update_from_paxos applying incremental 28995 2013-04-03 10:35:48.572262 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 v28995: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1951B/s wr, 0op/s 2013-04-03 10:35:48.572281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 send_pg_creates to 0 pgs 2013-04-03 10:35:48.572286 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 update_logger 2013-04-03 10:35:48.572296 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 create_pending v 28996 2013-04-03 10:35:48.572305 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 _updated_stats for osd.2 10.252.24.30:6800/1508 2013-04-03 10:35:48.572322 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 check_osd_map already seen 602 >= 602 2013-04-03 10:35:48.572325 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 update_logger 2013-04-03 10:35:48.572336 7fcdffcdb700 0 log [INF] : pgmap v28995: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1951B/s wr, 0op/s 2013-04-03 10:35:48.572390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos 2013-04-03 10:35:48.572408 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 2013-04-03 10:35:48.572425 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:48.572467 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos 2013-04-03 10:35:48.572482 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 2013-04-03 10:35:48.572494 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:48.572506 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:35:48.572528 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:48.572541 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:35:48.572553 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 logging 2013-04-03 10:35:48.572337 mon.0 10.252.24.25:6789/0 57 : [INF] pgmap v28995: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1951B/s wr, 0op/s 2013-04-03 10:35:48.585552 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos 2013-04-03 10:35:48.585569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 2013-04-03 10:35:48.585585 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:48.924791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:48.924801 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x295d2c0 from 10.252.24.88:6789/0 2013-04-03 10:35:48.924851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 35 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:35:49.378846 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21691 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:35:49.572011 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 encode_pending v394001 2013-04-03 10:35:49.602719 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos 2013-04-03 10:35:49.602756 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394000 2013-04-03 10:35:49.602770 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos latest full 393798 2013-04-03 10:35:49.602789 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos applying incremental log 394001 2013-04-03 10:35:48.572337 mon.0 10.252.24.25:6789/0 57 : [INF] pgmap v28995: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1951B/s wr, 0op/s 2013-04-03 10:35:49.602840 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 check_subs 2013-04-03 10:35:49.602872 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 create_pending v 394002 2013-04-03 10:35:49.602894 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:35:49.602958 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos 2013-04-03 10:35:49.602974 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394001 2013-04-03 10:35:49.602992 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:49.628484 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos 2013-04-03 10:35:49.628504 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394001 2013-04-03 10:35:49.628521 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:51.579079 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140321 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:35:52.603534 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos 2013-04-03 10:35:52.603555 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394001 2013-04-03 10:35:52.603572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:52.924962 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:52.924973 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x295d000 from 10.252.24.88:6789/0 2013-04-03 10:35:52.925017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 36 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:35:53.328922 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 check_down_pgs 2013-04-03 10:35:53.328952 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 v28995: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1951B/s wr, 0op/s 2013-04-03 10:35:53.328979 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:35:53.329004 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:35:53.329041 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:35:53.329346 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos 2013-04-03 10:35:53.329370 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394001 2013-04-03 10:35:53.329380 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 log 2013-04-03 10:35:53.329397 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:53.329415 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:35:53.378944 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21692 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:35:54.564129 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:54.564139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:35:54.564190 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 preprocess_query pg_stats(1 pgs tid 14 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:35:54.564218 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 prepare_update pg_stats(1 pgs tid 14 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:35:54.564224 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 prepare_pg_stats pg_stats(1 pgs tid 14 v 602) v1 from osd.1 2013-04-03 10:35:54.564231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:35:54.614410 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 encode_pending v 28996 2013-04-03 10:35:54.644882 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 update_from_paxos applying incremental 28996 2013-04-03 10:35:54.644898 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 v28996: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s 2013-04-03 10:35:54.644919 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 send_pg_creates to 0 pgs 2013-04-03 10:35:54.644922 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 update_logger 2013-04-03 10:35:54.644932 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 create_pending v 28997 2013-04-03 10:35:54.644941 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:35:54.644956 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 check_osd_map already seen 602 >= 602 2013-04-03 10:35:54.644960 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 update_logger 2013-04-03 10:35:54.644971 7fcdffcdb700 0 log [INF] : pgmap v28996: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s 2013-04-03 10:35:54.645043 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos 2013-04-03 10:35:54.645063 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394001 2013-04-03 10:35:54.645082 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:54.645125 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos 2013-04-03 10:35:54.645140 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394001 2013-04-03 10:35:54.645149 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:54.645160 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:35:54.645181 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:35:54.645194 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:35:54.645205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 logging 2013-04-03 10:35:54.644972 mon.0 10.252.24.25:6789/0 58 : [INF] pgmap v28996: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s 2013-04-03 10:35:54.656910 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos 2013-04-03 10:35:54.656926 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394001 2013-04-03 10:35:54.656941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:55.579202 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140322 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:35:55.644628 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 encode_pending v394002 2013-04-03 10:35:55.674009 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos 2013-04-03 10:35:55.674035 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394001 2013-04-03 10:35:55.674050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos latest full 393798 2013-04-03 10:35:55.674069 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos applying incremental log 394002 2013-04-03 10:35:54.644972 mon.0 10.252.24.25:6789/0 58 : [INF] pgmap v28996: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s 2013-04-03 10:35:55.674114 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 check_subs 2013-04-03 10:35:55.674146 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 create_pending v 394003 2013-04-03 10:35:55.674169 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:35:55.674230 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos 2013-04-03 10:35:55.674246 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 2013-04-03 10:35:55.674263 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:55.685953 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos 2013-04-03 10:35:55.685973 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 2013-04-03 10:35:55.685988 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:56.925142 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:35:56.925153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29968c0 from 10.252.24.88:6789/0 2013-04-03 10:35:56.925204 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 37 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:35:57.379051 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21693 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:35:58.329557 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 check_down_pgs 2013-04-03 10:35:58.329593 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 v28996: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s 2013-04-03 10:35:58.329629 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:35:58.329654 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:35:58.329694 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:35:58.329725 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos 2013-04-03 10:35:58.329744 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 2013-04-03 10:35:58.329754 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 log 2013-04-03 10:35:58.329771 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:58.329790 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:35:58.674923 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos 2013-04-03 10:35:58.674943 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 2013-04-03 10:35:58.674960 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:35:59.579330 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140323 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:36:00.249202 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:32978/0) 2013-04-03 10:36:00.925170 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:00.925182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2996600 from 10.252.24.88:6789/0 2013-04-03 10:36:00.925232 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 38 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:36:01.379184 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21694 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:36:01.674893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos 2013-04-03 10:36:01.674913 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 2013-04-03 10:36:01.674930 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:36:03.329938 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 v28996: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s 2013-04-03 10:36:03.329975 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:36:03.329999 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:36:03.330042 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:36:03.330077 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos 2013-04-03 10:36:03.330096 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 2013-04-03 10:36:03.330106 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 log 2013-04-03 10:36:03.330123 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:36:03.330142 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:36:03.486318 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:03.486327 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:36:03.486362 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 preprocess_query pg_stats(1 pgs tid 7 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:36:03.486385 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 prepare_update pg_stats(1 pgs tid 7 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:36:03.486394 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 prepare_pg_stats pg_stats(1 pgs tid 7 v 602) v1 from osd.2 2013-04-03 10:36:03.486400 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:36:03.537065 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 encode_pending v 28997 2013-04-03 10:36:03.579449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140324 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:36:03.587434 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 update_from_paxos applying incremental 28997 2013-04-03 10:36:03.587450 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 v28997: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:03.587469 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 send_pg_creates to 0 pgs 2013-04-03 10:36:03.587473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 update_logger 2013-04-03 10:36:03.587483 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 create_pending v 28998 2013-04-03 10:36:03.587493 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 _updated_stats for osd.2 10.252.24.30:6800/1508 2013-04-03 10:36:03.587510 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 check_osd_map already seen 602 >= 602 2013-04-03 10:36:03.587514 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 update_logger 2013-04-03 10:36:03.587524 7fcdffcdb700 0 log [INF] : pgmap v28997: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:03.587588 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos 2013-04-03 10:36:03.587605 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 2013-04-03 10:36:03.587623 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:36:03.587665 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos 2013-04-03 10:36:03.587680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 2013-04-03 10:36:03.587690 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:03.587702 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:36:03.587724 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:03.587737 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:36:03.587748 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 logging 2013-04-03 10:36:03.587526 mon.0 10.252.24.25:6789/0 59 : [INF] pgmap v28997: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:03.600318 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos 2013-04-03 10:36:03.600335 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 2013-04-03 10:36:03.600351 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:36:04.587228 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 encode_pending v394003 2013-04-03 10:36:04.616290 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos 2013-04-03 10:36:04.616314 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos version 394003 summary v 394002 2013-04-03 10:36:04.616331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos latest full 393798 2013-04-03 10:36:04.616350 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos applying incremental log 394003 2013-04-03 10:36:03.587526 mon.0 10.252.24.25:6789/0 59 : [INF] pgmap v28997: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:04.616395 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 check_subs 2013-04-03 10:36:04.616426 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 create_pending v 394004 2013-04-03 10:36:04.616449 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:36:04.616512 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos 2013-04-03 10:36:04.616528 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos version 394003 summary v 394003 2013-04-03 10:36:04.616544 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:36:04.628663 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos 2013-04-03 10:36:04.628680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos version 394003 summary v 394003 2013-04-03 10:36:04.628699 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:36:04.925347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:04.925358 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29c58c0 from 10.252.24.88:6789/0 2013-04-03 10:36:04.925409 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 39 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:36:05.379369 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21695 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:36:07.579578 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140325 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:36:07.617213 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos 2013-04-03 10:36:07.617234 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos version 394003 summary v 394003 2013-04-03 10:36:07.617252 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:36:08.330291 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 check_down_pgs 2013-04-03 10:36:08.330322 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 v28997: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:08.330351 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:36:08.330376 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:36:08.330413 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:36:08.330448 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos 2013-04-03 10:36:08.330467 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos version 394003 summary v 394003 2013-04-03 10:36:08.330477 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 log 2013-04-03 10:36:08.330494 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:36:08.330513 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:36:08.627484 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) 2013-04-03 10:36:08.627511 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader 2013-04-03 10:36:08.925494 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:08.925504 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c5600 from 10.252.24.88:6789/0 2013-04-03 10:36:08.925557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 40 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:36:09.070840 7fcdfe8d5700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.30:6789/0 mon protocol 2 2013-04-03 10:36:09.070944 7fcdfe8d5700 0 -- 10.252.24.25:6789/0 >> 10.252.24.30:6789/0 pipe(0x29b4500 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state connecting 2013-04-03 10:36:09.087280 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:36:09.087287 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:36:09.379416 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21696 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:36:09.565931 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:09.565940 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:36:09.565973 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28997 preprocess_query pg_stats(1 pgs tid 15 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:36:09.565996 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28997 prepare_update pg_stats(1 pgs tid 15 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:36:09.566001 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28997 prepare_pg_stats pg_stats(1 pgs tid 15 v 602) v1 from osd.1 2013-04-03 10:36:09.566007 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28997 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:36:09.616187 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28997 encode_pending v 28998 2013-04-03 10:36:09.648889 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28997 update_from_paxos applying incremental 28998 2013-04-03 10:36:09.648904 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 v28998: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:09.648923 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 send_pg_creates to 0 pgs 2013-04-03 10:36:09.648927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 update_logger 2013-04-03 10:36:09.648937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 create_pending v 28999 2013-04-03 10:36:09.648945 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28998 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:36:09.648961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 check_osd_map already seen 602 >= 602 2013-04-03 10:36:09.648964 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 update_logger 2013-04-03 10:36:09.648974 7fcdffcdb700 0 log [INF] : pgmap v28998: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:09.649031 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 update_from_paxos 2013-04-03 10:36:09.649050 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 update_from_paxos version 394003 summary v 394003 2013-04-03 10:36:09.649069 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:09.649143 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 update_from_paxos 2013-04-03 10:36:09.649158 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 update_from_paxos version 394003 summary v 394003 2013-04-03 10:36:09.649166 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:09.649178 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:36:09.649200 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:09.649212 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:36:09.649223 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 logging 2013-04-03 10:36:09.648975 mon.0 10.252.24.25:6789/0 60 : [INF] pgmap v28998: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:09.660434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 update_from_paxos 2013-04-03 10:36:09.660449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 update_from_paxos version 394003 summary v 394003 2013-04-03 10:36:09.660464 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:09.732867 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:09.732875 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:36:09.732899 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 preprocess_query pg_stats(1 pgs tid 30438 v 602) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:36:09.732918 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 prepare_update pg_stats(1 pgs tid 30438 v 602) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:36:09.732924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 prepare_pg_stats pg_stats(1 pgs tid 30438 v 602) v1 from osd.3 2013-04-03 10:36:09.732930 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:36:10.648653 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28998 encode_pending v 28999 2013-04-03 10:36:10.650467 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394003 encode_pending v394004 2013-04-03 10:36:10.676715 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28998 update_from_paxos applying incremental 28999 2013-04-03 10:36:10.676735 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 v28999: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s 2013-04-03 10:36:10.676754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 send_pg_creates to 0 pgs 2013-04-03 10:36:10.676757 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 update_logger 2013-04-03 10:36:10.676766 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 create_pending v 29000 2013-04-03 10:36:10.676775 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28999 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:36:10.676793 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 check_osd_map already seen 602 >= 602 2013-04-03 10:36:10.676796 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 update_logger 2013-04-03 10:36:10.676806 7fcdffcdb700 0 log [INF] : pgmap v28999: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s 2013-04-03 10:36:10.714503 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos 2013-04-03 10:36:10.714528 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394003 2013-04-03 10:36:10.714542 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos latest full 393798 2013-04-03 10:36:10.714561 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394004 update_from_paxos applying incremental log 394004 2013-04-03 10:36:09.648975 mon.0 10.252.24.25:6789/0 60 : [INF] pgmap v28998: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:10.714602 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 check_subs 2013-04-03 10:36:10.714632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 create_pending v 394005 2013-04-03 10:36:10.714653 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394004 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:36:10.714685 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos 2013-04-03 10:36:10.714700 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394004 2013-04-03 10:36:10.714708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:10.714719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:36:10.714734 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 nothing new 2013-04-03 10:36:10.714782 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos 2013-04-03 10:36:10.714797 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394004 2013-04-03 10:36:10.714814 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:10.726629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos 2013-04-03 10:36:10.726645 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394004 2013-04-03 10:36:10.726660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:11.579708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140326 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:36:12.925598 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:12.925608 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c4b00 from 10.252.24.88:6789/0 2013-04-03 10:36:12.925660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 41 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:36:13.330661 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28999 check_down_pgs 2013-04-03 10:36:13.330687 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28999 v28999: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s 2013-04-03 10:36:13.330713 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:36:13.330737 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:36:13.330774 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:36:13.330804 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos 2013-04-03 10:36:13.330823 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394004 2013-04-03 10:36:13.330832 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394004 log 2013-04-03 10:36:13.330848 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:13.330866 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:36:13.379510 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21697 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:36:13.487546 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:13.487554 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:36:13.487583 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 preprocess_query pg_stats(1 pgs tid 8 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:36:13.487604 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 prepare_update pg_stats(1 pgs tid 8 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:36:13.487609 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 prepare_pg_stats pg_stats(1 pgs tid 8 v 602) v1 from osd.2 2013-04-03 10:36:13.487615 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:36:13.537793 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28999 encode_pending v 29000 2013-04-03 10:36:13.583007 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28999 update_from_paxos applying incremental 29000 2013-04-03 10:36:13.583023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 v29000: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3394B/s wr, 0op/s 2013-04-03 10:36:13.583039 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 send_pg_creates to 0 pgs 2013-04-03 10:36:13.583042 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 update_logger 2013-04-03 10:36:13.583051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 create_pending v 29001 2013-04-03 10:36:13.583059 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29000 _updated_stats for osd.2 10.252.24.30:6800/1508 2013-04-03 10:36:13.583079 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 check_osd_map already seen 602 >= 602 2013-04-03 10:36:13.583082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 update_logger 2013-04-03 10:36:13.583092 7fcdffcdb700 0 log [INF] : pgmap v29000: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3394B/s wr, 0op/s 2013-04-03 10:36:13.583153 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos 2013-04-03 10:36:13.583171 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394004 2013-04-03 10:36:13.583187 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:13.583229 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos 2013-04-03 10:36:13.583244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394004 2013-04-03 10:36:13.583253 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:13.583265 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:36:13.583292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:13.583304 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:36:13.583315 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 logging 2013-04-03 10:36:10.676807 mon.0 10.252.24.25:6789/0 61 : [INF] pgmap v28999: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s 2013-04-03 10:36:13.583338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 logging 2013-04-03 10:36:13.583093 mon.0 10.252.24.25:6789/0 62 : [INF] pgmap v29000: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3394B/s wr, 0op/s 2013-04-03 10:36:13.594536 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos 2013-04-03 10:36:13.594552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394004 2013-04-03 10:36:13.594567 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:14.582803 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394004 encode_pending v394005 2013-04-03 10:36:14.610772 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos 2013-04-03 10:36:14.610799 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394004 2013-04-03 10:36:14.610815 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos latest full 393798 2013-04-03 10:36:14.610834 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394005 update_from_paxos applying incremental log 394005 2013-04-03 10:36:10.676807 mon.0 10.252.24.25:6789/0 61 : [INF] pgmap v28999: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s 2013-04-03 10:36:14.610861 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394005 update_from_paxos applying incremental log 394005 2013-04-03 10:36:13.583093 mon.0 10.252.24.25:6789/0 62 : [INF] pgmap v29000: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3394B/s wr, 0op/s 2013-04-03 10:36:14.610898 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 check_subs 2013-04-03 10:36:14.610930 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 create_pending v 394006 2013-04-03 10:36:14.610952 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394005 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:36:14.611014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos 2013-04-03 10:36:14.611030 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 2013-04-03 10:36:14.611050 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:14.622395 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos 2013-04-03 10:36:14.622411 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 2013-04-03 10:36:14.622425 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:15.579830 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140327 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:36:16.925747 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:16.925758 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c4580 from 10.252.24.88:6789/0 2013-04-03 10:36:16.925809 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 42 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:36:17.379679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21698 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:36:17.414630 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 2013-04-03 10:36:17.414637 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:36:17.611831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos 2013-04-03 10:36:17.611854 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 2013-04-03 10:36:17.611871 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:18.331011 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29000 check_down_pgs 2013-04-03 10:36:18.331042 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29000 v29000: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3394B/s wr, 0op/s 2013-04-03 10:36:18.331069 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:36:18.331093 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:36:18.331129 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:36:18.331161 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos 2013-04-03 10:36:18.331179 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 2013-04-03 10:36:18.331188 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394005 log 2013-04-03 10:36:18.331205 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:18.331223 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:36:19.579962 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140328 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:36:20.611880 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos 2013-04-03 10:36:20.611904 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 2013-04-03 10:36:20.611922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:20.925820 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:20.925828 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c42c0 from 10.252.24.88:6789/0 2013-04-03 10:36:20.925864 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 43 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:36:21.379839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21699 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:36:23.331365 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29000 v29000: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3394B/s wr, 0op/s 2013-04-03 10:36:23.331406 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:36:23.331428 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:36:23.331469 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:36:23.331499 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos 2013-04-03 10:36:23.331517 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 2013-04-03 10:36:23.331526 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394005 log 2013-04-03 10:36:23.331543 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:23.331565 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:36:23.580076 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140329 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:36:23.611860 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos 2013-04-03 10:36:23.611879 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 2013-04-03 10:36:23.611895 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:24.567592 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:24.567601 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:36:24.567652 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 preprocess_query pg_stats(1 pgs tid 16 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:36:24.567675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 prepare_update pg_stats(1 pgs tid 16 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:36:24.567681 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 prepare_pg_stats pg_stats(1 pgs tid 16 v 602) v1 from osd.1 2013-04-03 10:36:24.567687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:36:24.617874 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29000 encode_pending v 29001 2013-04-03 10:36:24.657210 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29000 update_from_paxos applying incremental 29001 2013-04-03 10:36:24.657226 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 v29001: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s 2013-04-03 10:36:24.657245 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 send_pg_creates to 0 pgs 2013-04-03 10:36:24.657248 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 update_logger 2013-04-03 10:36:24.657257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 create_pending v 29002 2013-04-03 10:36:24.657266 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29001 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:36:24.657282 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 check_osd_map already seen 602 >= 602 2013-04-03 10:36:24.657285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 update_logger 2013-04-03 10:36:24.657295 7fcdffcdb700 0 log [INF] : pgmap v29001: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s 2013-04-03 10:36:24.657362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos 2013-04-03 10:36:24.657382 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 2013-04-03 10:36:24.657401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:24.657444 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos 2013-04-03 10:36:24.657459 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 2013-04-03 10:36:24.657468 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:24.657478 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:36:24.657500 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:24.657512 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:36:24.657521 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 logging 2013-04-03 10:36:24.657296 mon.0 10.252.24.25:6789/0 63 : [INF] pgmap v29001: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s 2013-04-03 10:36:24.668033 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos 2013-04-03 10:36:24.668049 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 2013-04-03 10:36:24.668063 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:24.925935 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:24.925943 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29f5b80 from 10.252.24.88:6789/0 2013-04-03 10:36:24.925974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 44 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:36:25.380010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21700 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:36:25.479045 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 2013-04-03 10:36:25.479053 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:36:25.646612 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394005 encode_pending v394006 2013-04-03 10:36:25.766433 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos 2013-04-03 10:36:25.766468 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos version 394006 summary v 394005 2013-04-03 10:36:25.766482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos latest full 393798 2013-04-03 10:36:25.766500 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394006 update_from_paxos applying incremental log 394006 2013-04-03 10:36:24.657296 mon.0 10.252.24.25:6789/0 63 : [INF] pgmap v29001: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s 2013-04-03 10:36:25.766551 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 check_subs 2013-04-03 10:36:25.766582 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 create_pending v 394007 2013-04-03 10:36:25.766603 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394006 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:36:25.766666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos 2013-04-03 10:36:25.766681 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos version 394006 summary v 394006 2013-04-03 10:36:25.766702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:25.779194 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos 2013-04-03 10:36:25.779211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos version 394006 summary v 394006 2013-04-03 10:36:25.779225 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:27.580198 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140330 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:36:28.331711 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29001 check_down_pgs 2013-04-03 10:36:28.331735 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29001 v29001: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s 2013-04-03 10:36:28.331761 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:36:28.331785 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:36:28.331821 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:36:28.331851 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos 2013-04-03 10:36:28.331869 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos version 394006 summary v 394006 2013-04-03 10:36:28.331879 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394006 log 2013-04-03 10:36:28.331895 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:28.331917 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:36:28.489587 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:28.489596 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:36:28.489631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 preprocess_query pg_stats(1 pgs tid 9 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:36:28.489653 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 prepare_update pg_stats(1 pgs tid 9 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:36:28.489658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 prepare_pg_stats pg_stats(1 pgs tid 9 v 602) v1 from osd.2 2013-04-03 10:36:28.489664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:36:28.539849 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29001 encode_pending v 29002 2013-04-03 10:36:28.573257 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29001 update_from_paxos applying incremental 29002 2013-04-03 10:36:28.573272 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 v29002: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:28.573290 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 send_pg_creates to 0 pgs 2013-04-03 10:36:28.573294 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 update_logger 2013-04-03 10:36:28.573304 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 create_pending v 29003 2013-04-03 10:36:28.573313 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29002 _updated_stats for osd.2 10.252.24.30:6800/1508 2013-04-03 10:36:28.573328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 check_osd_map already seen 602 >= 602 2013-04-03 10:36:28.573331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 update_logger 2013-04-03 10:36:28.573341 7fcdffcdb700 0 log [INF] : pgmap v29002: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:28.573394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos 2013-04-03 10:36:28.573411 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos version 394006 summary v 394006 2013-04-03 10:36:28.573428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:28.573472 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos 2013-04-03 10:36:28.573486 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos version 394006 summary v 394006 2013-04-03 10:36:28.573494 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:28.573504 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:36:28.573524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:28.573535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:36:28.573546 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 logging 2013-04-03 10:36:28.573342 mon.0 10.252.24.25:6789/0 64 : [INF] pgmap v29002: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:28.588917 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos 2013-04-03 10:36:28.588933 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos version 394006 summary v 394006 2013-04-03 10:36:28.588947 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:28.926115 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:28.926123 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29f5600 from 10.252.24.88:6789/0 2013-04-03 10:36:28.926154 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 45 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:36:29.380077 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21701 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:36:29.567486 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394006 encode_pending v394007 2013-04-03 10:36:29.596789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos 2013-04-03 10:36:29.596817 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394006 2013-04-03 10:36:29.596831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos latest full 393798 2013-04-03 10:36:29.596849 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394007 update_from_paxos applying incremental log 394007 2013-04-03 10:36:28.573342 mon.0 10.252.24.25:6789/0 64 : [INF] pgmap v29002: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:29.596894 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 check_subs 2013-04-03 10:36:29.596924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 create_pending v 394008 2013-04-03 10:36:29.596946 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394007 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:36:29.597008 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos 2013-04-03 10:36:29.597023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 2013-04-03 10:36:29.597041 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:29.621675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos 2013-04-03 10:36:29.621692 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 2013-04-03 10:36:29.621706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:31.580319 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140331 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:36:32.597702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos 2013-04-03 10:36:32.597722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 2013-04-03 10:36:32.597738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:32.926220 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:32.926229 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29f5340 from 10.252.24.88:6789/0 2013-04-03 10:36:32.926263 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 46 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:36:33.332024 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29002 check_down_pgs 2013-04-03 10:36:33.332052 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29002 v29002: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:33.332078 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:36:33.332102 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:36:33.332136 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:36:33.332166 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos 2013-04-03 10:36:33.332185 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 2013-04-03 10:36:33.332194 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394007 log 2013-04-03 10:36:33.332210 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:33.332231 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:36:33.380152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21702 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:36:35.580445 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140332 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:36:35.597714 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos 2013-04-03 10:36:35.597733 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 2013-04-03 10:36:35.597749 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:36.926349 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:36.926358 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29f5080 from 10.252.24.88:6789/0 2013-04-03 10:36:36.926409 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 47 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:36:37.380294 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21703 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:36:38.332378 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29002 v29002: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:38.332417 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:36:38.332439 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:36:38.332479 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:36:38.332510 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos 2013-04-03 10:36:38.332528 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 2013-04-03 10:36:38.332537 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394007 log 2013-04-03 10:36:38.332554 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:38.332572 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:36:38.597806 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos 2013-04-03 10:36:38.597824 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 2013-04-03 10:36:38.597839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:39.569250 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:39.569259 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:36:39.569309 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 preprocess_query pg_stats(1 pgs tid 17 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:36:39.569333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 prepare_update pg_stats(1 pgs tid 17 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:36:39.569338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 prepare_pg_stats pg_stats(1 pgs tid 17 v 602) v1 from osd.1 2013-04-03 10:36:39.569345 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:36:39.580516 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140333 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:36:39.619525 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29002 encode_pending v 29003 2013-04-03 10:36:39.664170 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29002 update_from_paxos applying incremental 29003 2013-04-03 10:36:39.664185 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 v29003: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:39.664202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 send_pg_creates to 0 pgs 2013-04-03 10:36:39.664205 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 update_logger 2013-04-03 10:36:39.664214 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 create_pending v 29004 2013-04-03 10:36:39.664222 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29003 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:36:39.664239 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 check_osd_map already seen 602 >= 602 2013-04-03 10:36:39.664241 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 update_logger 2013-04-03 10:36:39.664251 7fcdffcdb700 0 log [INF] : pgmap v29003: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:39.664306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos 2013-04-03 10:36:39.664325 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 2013-04-03 10:36:39.664341 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:39.664383 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos 2013-04-03 10:36:39.664397 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 2013-04-03 10:36:39.664405 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:39.664415 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:36:39.664435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:39.664446 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:36:39.664456 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 logging 2013-04-03 10:36:39.664253 mon.0 10.252.24.25:6789/0 65 : [INF] pgmap v29003: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:39.675794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos 2013-04-03 10:36:39.675810 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 2013-04-03 10:36:39.675824 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:39.735845 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:39.735852 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 2013-04-03 10:36:39.735877 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 preprocess_query pg_stats(1 pgs tid 30439 v 602) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:36:39.735897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 prepare_update pg_stats(1 pgs tid 30439 v 602) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:36:39.735903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 prepare_pg_stats pg_stats(1 pgs tid 30439 v 602) v1 from osd.3 2013-04-03 10:36:39.735907 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:36:40.663953 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29003 encode_pending v 29004 2013-04-03 10:36:40.665240 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394007 encode_pending v394008 2013-04-03 10:36:40.708681 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29003 update_from_paxos applying incremental 29004 2013-04-03 10:36:40.708699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 v29004: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 422B/s wr, 0op/s 2013-04-03 10:36:40.708719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 send_pg_creates to 0 pgs 2013-04-03 10:36:40.708722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 update_logger 2013-04-03 10:36:40.708731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 create_pending v 29005 2013-04-03 10:36:40.708740 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29004 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:36:40.708758 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 check_osd_map already seen 602 >= 602 2013-04-03 10:36:40.708760 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 update_logger 2013-04-03 10:36:40.708772 7fcdffcdb700 0 log [INF] : pgmap v29004: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 422B/s wr, 0op/s 2013-04-03 10:36:40.745693 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos 2013-04-03 10:36:40.745716 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394007 2013-04-03 10:36:40.745730 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos latest full 393798 2013-04-03 10:36:40.745745 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394008 update_from_paxos applying incremental log 394008 2013-04-03 10:36:39.664253 mon.0 10.252.24.25:6789/0 65 : [INF] pgmap v29003: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:36:40.745790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 check_subs 2013-04-03 10:36:40.745827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 create_pending v 394009 2013-04-03 10:36:40.745850 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394008 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:36:40.746157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos 2013-04-03 10:36:40.746175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394008 2013-04-03 10:36:40.746183 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:40.746194 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:36:40.746210 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 nothing new 2013-04-03 10:36:40.746258 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos 2013-04-03 10:36:40.746273 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394008 2013-04-03 10:36:40.746289 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:40.758712 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos 2013-04-03 10:36:40.758728 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394008 2013-04-03 10:36:40.758742 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:40.926437 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:40.926445 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29f4840 from 10.252.24.88:6789/0 2013-04-03 10:36:40.926476 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 48 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:36:41.380401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21704 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:36:43.332720 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29004 check_down_pgs 2013-04-03 10:36:43.332747 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29004 v29004: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 422B/s wr, 0op/s 2013-04-03 10:36:43.332774 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:36:43.332796 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:36:43.332832 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:36:43.332862 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos 2013-04-03 10:36:43.332881 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394008 2013-04-03 10:36:43.332890 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394008 log 2013-04-03 10:36:43.332906 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:43.332925 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:36:43.491447 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:43.491456 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 2013-04-03 10:36:43.491496 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 preprocess_query pg_stats(1 pgs tid 10 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:36:43.491518 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 prepare_update pg_stats(1 pgs tid 10 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:36:43.491524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 prepare_pg_stats pg_stats(1 pgs tid 10 v 602) v1 from osd.2 2013-04-03 10:36:43.491530 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:36:43.541707 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29004 encode_pending v 29005 2013-04-03 10:36:43.572816 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29004 update_from_paxos applying incremental 29005 2013-04-03 10:36:43.572839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 v29005: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 4438B/s wr, 0op/s 2013-04-03 10:36:43.572857 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 send_pg_creates to 0 pgs 2013-04-03 10:36:43.572861 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 update_logger 2013-04-03 10:36:43.572872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 create_pending v 29006 2013-04-03 10:36:43.572880 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29005 _updated_stats for osd.2 10.252.24.30:6800/1508 2013-04-03 10:36:43.573090 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 check_osd_map already seen 602 >= 602 2013-04-03 10:36:43.573094 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 update_logger 2013-04-03 10:36:43.573105 7fcdffcdb700 0 log [INF] : pgmap v29005: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 4438B/s wr, 0op/s 2013-04-03 10:36:43.573167 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos 2013-04-03 10:36:43.573184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394008 2013-04-03 10:36:43.573202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:43.573244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos 2013-04-03 10:36:43.573263 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394008 2013-04-03 10:36:43.573272 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:43.573282 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:36:43.573308 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:43.573320 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:36:43.573330 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 logging 2013-04-03 10:36:40.708773 mon.0 10.252.24.25:6789/0 66 : [INF] pgmap v29004: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 422B/s wr, 0op/s 2013-04-03 10:36:43.573350 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 logging 2013-04-03 10:36:43.573106 mon.0 10.252.24.25:6789/0 67 : [INF] pgmap v29005: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 4438B/s wr, 0op/s 2013-04-03 10:36:43.580660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140334 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:36:43.597677 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos 2013-04-03 10:36:43.597694 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394008 2013-04-03 10:36:43.597709 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:44.572579 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394008 encode_pending v394009 2013-04-03 10:36:44.597130 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos 2013-04-03 10:36:44.597156 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394008 2013-04-03 10:36:44.597174 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos latest full 393798 2013-04-03 10:36:44.597190 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394009 update_from_paxos applying incremental log 394009 2013-04-03 10:36:40.708773 mon.0 10.252.24.25:6789/0 66 : [INF] pgmap v29004: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 422B/s wr, 0op/s 2013-04-03 10:36:44.597217 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394009 update_from_paxos applying incremental log 394009 2013-04-03 10:36:43.573106 mon.0 10.252.24.25:6789/0 67 : [INF] pgmap v29005: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 4438B/s wr, 0op/s 2013-04-03 10:36:44.597256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 check_subs 2013-04-03 10:36:44.597286 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 create_pending v 394010 2013-04-03 10:36:44.597308 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394009 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:36:44.597371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos 2013-04-03 10:36:44.597387 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394009 2013-04-03 10:36:44.597403 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:44.608753 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos 2013-04-03 10:36:44.608769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394009 2013-04-03 10:36:44.608783 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:44.926581 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:44.926590 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29f42c0 from 10.252.24.88:6789/0 2013-04-03 10:36:44.926622 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 49 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:36:45.380477 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21705 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:36:47.580817 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140335 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:36:47.598008 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos 2013-04-03 10:36:47.598028 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394009 2013-04-03 10:36:47.598044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:48.333076 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29005 check_down_pgs 2013-04-03 10:36:48.333107 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29005 v29005: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 4438B/s wr, 0op/s 2013-04-03 10:36:48.333134 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:36:48.333156 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:36:48.333193 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:36:48.333223 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos 2013-04-03 10:36:48.333242 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394009 2013-04-03 10:36:48.333251 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394009 log 2013-04-03 10:36:48.333268 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:48.333286 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:36:48.431214 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:48.431224 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:36:48.431274 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 preprocess_query pg_stats(0 pgs tid 58 v 602) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:36:48.431297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 prepare_update pg_stats(0 pgs tid 58 v 602) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:36:48.431308 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 prepare_pg_stats pg_stats(0 pgs tid 58 v 602) v1 from osd.4 2013-04-03 10:36:48.431314 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 got osd.4 osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1051 MB used, 56606 MB avail, 59728 MB total, peers []/[])) 2013-04-03 10:36:48.481461 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29005 encode_pending v 29006 2013-04-03 10:36:48.522636 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29005 update_from_paxos applying incremental 29006 2013-04-03 10:36:48.522646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 v29006: 240 pgs: 240 active+clean; 253 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 2095B/s wr, 0op/s 2013-04-03 10:36:48.522662 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 send_pg_creates to 0 pgs 2013-04-03 10:36:48.522664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 update_logger 2013-04-03 10:36:48.522673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 create_pending v 29007 2013-04-03 10:36:48.522682 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29006 _updated_stats for osd.4 10.252.24.32:6800/13473 2013-04-03 10:36:48.522697 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 check_osd_map already seen 602 >= 602 2013-04-03 10:36:48.522699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 update_logger 2013-04-03 10:36:48.522709 7fcdffcdb700 0 log [INF] : pgmap v29006: 240 pgs: 240 active+clean; 253 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 2095B/s wr, 0op/s 2013-04-03 10:36:48.522776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos 2013-04-03 10:36:48.522796 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394009 2013-04-03 10:36:48.522814 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:48.522857 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos 2013-04-03 10:36:48.522872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394009 2013-04-03 10:36:48.522881 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:48.522891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:36:48.522911 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:48.522923 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:36:48.522933 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 logging 2013-04-03 10:36:48.522710 mon.0 10.252.24.25:6789/0 68 : [INF] pgmap v29006: 240 pgs: 240 active+clean; 253 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 2095B/s wr, 0op/s 2013-04-03 10:36:48.534373 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos 2013-04-03 10:36:48.534390 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394009 2013-04-03 10:36:48.534404 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:48.926724 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:48.926732 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x295d580 from 10.252.24.88:6789/0 2013-04-03 10:36:48.926766 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 50 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:36:49.380627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21706 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:36:49.514286 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394009 encode_pending v394010 2013-04-03 10:36:49.542220 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos 2013-04-03 10:36:49.542246 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos version 394010 summary v 394009 2013-04-03 10:36:49.542260 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos latest full 393798 2013-04-03 10:36:49.542276 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394010 update_from_paxos applying incremental log 394010 2013-04-03 10:36:48.522710 mon.0 10.252.24.25:6789/0 68 : [INF] pgmap v29006: 240 pgs: 240 active+clean; 253 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 2095B/s wr, 0op/s 2013-04-03 10:36:49.542323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 check_subs 2013-04-03 10:36:49.542354 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 create_pending v 394011 2013-04-03 10:36:49.542375 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394010 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:36:49.542437 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos 2013-04-03 10:36:49.542452 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos version 394010 summary v 394010 2013-04-03 10:36:49.542470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:49.553883 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos 2013-04-03 10:36:49.553899 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos version 394010 summary v 394010 2013-04-03 10:36:49.553913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:49.800649 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:49.800658 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:36:49.800687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 preprocess_query pg_stats(0 pgs tid 407 v 602) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:36:49.800708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 prepare_update pg_stats(0 pgs tid 407 v 602) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:36:49.800713 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 prepare_pg_stats pg_stats(0 pgs tid 407 v 602) v1 from osd.0 2013-04-03 10:36:49.800719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 got osd.0 osd_stat(1139 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1138 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:36:50.329299 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:32985/0) 2013-04-03 10:36:50.542071 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29006 encode_pending v 29007 2013-04-03 10:36:50.711917 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29006 update_from_paxos applying incremental 29007 2013-04-03 10:36:50.711929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 v29007: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:36:50.711946 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 send_pg_creates to 0 pgs 2013-04-03 10:36:50.711950 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 update_logger 2013-04-03 10:36:50.711960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 create_pending v 29008 2013-04-03 10:36:50.711968 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29007 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:36:50.711985 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 check_osd_map already seen 602 >= 602 2013-04-03 10:36:50.711987 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 update_logger 2013-04-03 10:36:50.711997 7fcdffcdb700 0 log [INF] : pgmap v29007: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:36:50.712063 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos 2013-04-03 10:36:50.712083 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos version 394010 summary v 394010 2013-04-03 10:36:50.712099 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:50.712143 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos 2013-04-03 10:36:50.712158 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos version 394010 summary v 394010 2013-04-03 10:36:50.712165 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:50.712175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:36:50.712195 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:50.712207 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:36:50.712216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 logging 2013-04-03 10:36:50.711998 mon.0 10.252.24.25:6789/0 69 : [INF] pgmap v29007: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:36:50.724588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos 2013-04-03 10:36:50.724603 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos version 394010 summary v 394010 2013-04-03 10:36:50.724617 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:51.580950 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140336 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:36:51.711958 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394010 encode_pending v394011 2013-04-03 10:36:51.740695 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos 2013-04-03 10:36:51.740721 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos version 394011 summary v 394010 2013-04-03 10:36:51.740738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos latest full 393798 2013-04-03 10:36:51.740757 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394011 update_from_paxos applying incremental log 394011 2013-04-03 10:36:50.711998 mon.0 10.252.24.25:6789/0 69 : [INF] pgmap v29007: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:36:51.740800 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 check_subs 2013-04-03 10:36:51.740830 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 create_pending v 394012 2013-04-03 10:36:51.740852 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394011 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:36:51.740912 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos 2013-04-03 10:36:51.740927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos version 394011 summary v 394011 2013-04-03 10:36:51.740945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:51.752357 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos 2013-04-03 10:36:51.752373 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos version 394011 summary v 394011 2013-04-03 10:36:51.752387 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:52.926871 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:52.926881 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c5b80 from 10.252.24.88:6789/0 2013-04-03 10:36:52.926927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 51 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:36:53.333433 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29007 check_down_pgs 2013-04-03 10:36:53.333459 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29007 v29007: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:36:53.333484 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:36:53.333507 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:36:53.333544 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:36:53.333574 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos 2013-04-03 10:36:53.333592 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos version 394011 summary v 394011 2013-04-03 10:36:53.333602 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394011 log 2013-04-03 10:36:53.333621 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:53.333639 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:36:53.380674 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21707 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:36:54.570902 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:54.570911 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:36:54.570961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 preprocess_query pg_stats(1 pgs tid 18 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:36:54.570984 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 prepare_update pg_stats(1 pgs tid 18 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:36:54.570990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 prepare_pg_stats pg_stats(1 pgs tid 18 v 602) v1 from osd.1 2013-04-03 10:36:54.570997 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:36:54.621178 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29007 encode_pending v 29008 2013-04-03 10:36:54.655759 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29007 update_from_paxos applying incremental 29008 2013-04-03 10:36:54.655774 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 v29008: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 667B/s wr, 0op/s 2013-04-03 10:36:54.655792 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 send_pg_creates to 0 pgs 2013-04-03 10:36:54.655796 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 update_logger 2013-04-03 10:36:54.655809 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 create_pending v 29009 2013-04-03 10:36:54.655818 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29008 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:36:54.655833 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 check_osd_map already seen 602 >= 602 2013-04-03 10:36:54.655835 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 update_logger 2013-04-03 10:36:54.655845 7fcdffcdb700 0 log [INF] : pgmap v29008: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 667B/s wr, 0op/s 2013-04-03 10:36:54.655901 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos 2013-04-03 10:36:54.655919 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos version 394011 summary v 394011 2013-04-03 10:36:54.655937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:54.655980 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos 2013-04-03 10:36:54.655994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos version 394011 summary v 394011 2013-04-03 10:36:54.656003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:54.656014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:36:54.656034 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:36:54.656049 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:36:54.656058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 logging 2013-04-03 10:36:54.655846 mon.0 10.252.24.25:6789/0 70 : [INF] pgmap v29008: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 667B/s wr, 0op/s 2013-04-03 10:36:54.670971 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos 2013-04-03 10:36:54.670986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos version 394011 summary v 394011 2013-04-03 10:36:54.671000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:55.581039 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140337 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:36:55.649517 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394011 encode_pending v394012 2013-04-03 10:36:55.853686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos 2013-04-03 10:36:55.853712 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394011 2013-04-03 10:36:55.853726 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos latest full 393798 2013-04-03 10:36:55.853742 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394012 update_from_paxos applying incremental log 394012 2013-04-03 10:36:54.655846 mon.0 10.252.24.25:6789/0 70 : [INF] pgmap v29008: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 667B/s wr, 0op/s 2013-04-03 10:36:55.853788 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 check_subs 2013-04-03 10:36:55.853826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 create_pending v 394013 2013-04-03 10:36:55.853849 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394012 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:36:55.890816 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos 2013-04-03 10:36:55.890833 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 2013-04-03 10:36:55.890849 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:55.902418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos 2013-04-03 10:36:55.902435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 2013-04-03 10:36:55.902449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:56.927015 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:36:56.927025 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x273a2c0 from 10.252.24.88:6789/0 2013-04-03 10:36:56.927075 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 52 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:36:57.380786 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21708 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:36:58.333790 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29008 check_down_pgs 2013-04-03 10:36:58.333819 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29008 v29008: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 667B/s wr, 0op/s 2013-04-03 10:36:58.333846 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:36:58.333869 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:36:58.333905 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:36:58.333935 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos 2013-04-03 10:36:58.333953 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 2013-04-03 10:36:58.333963 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394012 log 2013-04-03 10:36:58.333979 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:58.334002 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:36:58.891639 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos 2013-04-03 10:36:58.891658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 2013-04-03 10:36:58.891675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:36:59.581195 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140338 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:37:00.927102 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:00.927112 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x273a580 from 10.252.24.88:6789/0 2013-04-03 10:37:00.927163 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 53 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:37:01.380898 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21709 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:37:01.891702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos 2013-04-03 10:37:01.891721 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 2013-04-03 10:37:01.891738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:03.334127 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29008 v29008: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 667B/s wr, 0op/s 2013-04-03 10:37:03.334166 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:37:03.334189 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:37:03.334230 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:37:03.334260 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos 2013-04-03 10:37:03.334279 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 2013-04-03 10:37:03.334286 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394012 log 2013-04-03 10:37:03.334303 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:03.334321 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:37:03.493862 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:03.493872 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:37:03.493924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 preprocess_query pg_stats(1 pgs tid 11 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:37:03.493947 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 prepare_update pg_stats(1 pgs tid 11 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:37:03.493953 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 prepare_pg_stats pg_stats(1 pgs tid 11 v 602) v1 from osd.2 2013-04-03 10:37:03.493959 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:37:03.544157 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29008 encode_pending v 29009 2013-04-03 10:37:03.580517 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29008 update_from_paxos applying incremental 29009 2013-04-03 10:37:03.580538 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29009 v29009: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 6300B/s wr, 0op/s 2013-04-03 10:37:03.580559 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29009 send_pg_creates to 0 pgs 2013-04-03 10:37:03.580561 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29009 update_logger 2013-04-03 10:37:03.580571 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29009 create_pending v 29010 2013-04-03 10:37:03.580579 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29009 _updated_stats for osd.2 10.252.24.30:6800/1508 2013-04-03 10:37:03.580598 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29009 check_osd_map already seen 602 >= 602 2013-04-03 10:37:03.580600 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29009 update_logger 2013-04-03 10:37:03.580610 7fcdffcdb700 0 log [INF] : pgmap v29009: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 6300B/s wr, 0op/s 2013-04-03 10:37:03.580669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos 2013-04-03 10:37:03.580688 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 2013-04-03 10:37:03.580707 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:03.580751 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos 2013-04-03 10:37:03.580765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 2013-04-03 10:37:03.580774 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:03.580789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:37:03.580810 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:03.580822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:37:03.580831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 logging 2013-04-03 10:37:03.580612 mon.0 10.252.24.25:6789/0 71 : [INF] pgmap v29009: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 6300B/s wr, 0op/s 2013-04-03 10:37:03.581256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140339 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:37:03.631205 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos 2013-04-03 10:37:03.631223 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 2013-04-03 10:37:03.631238 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:04.579027 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394012 encode_pending v394013 2013-04-03 10:37:04.609843 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos 2013-04-03 10:37:04.609870 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos version 394013 summary v 394012 2013-04-03 10:37:04.609884 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos latest full 393798 2013-04-03 10:37:04.609900 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394013 update_from_paxos applying incremental log 394013 2013-04-03 10:37:03.580612 mon.0 10.252.24.25:6789/0 71 : [INF] pgmap v29009: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 6300B/s wr, 0op/s 2013-04-03 10:37:04.609962 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 check_subs 2013-04-03 10:37:04.609995 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 create_pending v 394014 2013-04-03 10:37:04.610017 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394013 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:04.610080 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos 2013-04-03 10:37:04.610096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos version 394013 summary v 394013 2013-04-03 10:37:04.610112 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:04.621834 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos 2013-04-03 10:37:04.621850 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos version 394013 summary v 394013 2013-04-03 10:37:04.621865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:04.927219 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:04.927227 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2996b80 from 10.252.24.88:6789/0 2013-04-03 10:37:04.927259 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 54 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:37:05.381003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21710 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:37:07.581451 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140340 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:37:07.610675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos 2013-04-03 10:37:07.610695 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos version 394013 summary v 394013 2013-04-03 10:37:07.610712 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:08.334466 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29009 check_down_pgs 2013-04-03 10:37:08.334496 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29009 v29009: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 6300B/s wr, 0op/s 2013-04-03 10:37:08.334523 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:37:08.334545 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:37:08.334581 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 2013-04-03 10:37:08.334612 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos 2013-04-03 10:37:08.334630 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos version 394013 summary v 394013 2013-04-03 10:37:08.334639 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394013 log 2013-04-03 10:37:08.334656 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:08.334674 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:37:08.927379 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:08.927388 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2995000 from 10.252.24.88:6789/0 2013-04-03 10:37:08.927439 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 55 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:37:09.193131 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:37:09.193138 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:37:09.193147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 2013-04-03 10:37:09.381095 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21711 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:37:09.573007 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:09.573018 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:37:09.573067 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29009 preprocess_query pg_stats(2 pgs tid 19 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:09.573091 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29009 prepare_update pg_stats(2 pgs tid 19 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:09.573097 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29009 prepare_pg_stats pg_stats(2 pgs tid 19 v 602) v1 from osd.1 2013-04-03 10:37:09.573104 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29009 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:37:09.623308 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29009 encode_pending v 29010 2013-04-03 10:37:09.722307 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29009 update_from_paxos applying incremental 29010 2013-04-03 10:37:09.722330 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 v29010: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 5460B/s wr, 0op/s 2013-04-03 10:37:09.722350 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 send_pg_creates to 0 pgs 2013-04-03 10:37:09.722354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 update_logger 2013-04-03 10:37:09.722363 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 create_pending v 29011 2013-04-03 10:37:09.722373 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:09.722392 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 check_osd_map already seen 602 >= 602 2013-04-03 10:37:09.722396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 update_logger 2013-04-03 10:37:09.722407 7fcdffcdb700 0 log [INF] : pgmap v29010: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 5460B/s wr, 0op/s 2013-04-03 10:37:09.722466 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 update_from_paxos 2013-04-03 10:37:09.722486 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 update_from_paxos version 394013 summary v 394013 2013-04-03 10:37:09.722505 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:09.722549 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 update_from_paxos 2013-04-03 10:37:09.722564 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 update_from_paxos version 394013 summary v 394013 2013-04-03 10:37:09.722573 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:09.722584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:37:09.722606 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:09.722618 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:37:09.722628 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 logging 2013-04-03 10:37:09.722408 mon.0 10.252.24.25:6789/0 72 : [INF] pgmap v29010: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 5460B/s wr, 0op/s 2013-04-03 10:37:09.749719 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 update_from_paxos 2013-04-03 10:37:09.749736 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 update_from_paxos version 394013 summary v 394013 2013-04-03 10:37:09.749751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:09.750042 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:09.750050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 2013-04-03 10:37:09.750076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 preprocess_query pg_stats(1 pgs tid 30440 v 602) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:09.750097 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 prepare_update pg_stats(1 pgs tid 30440 v 602) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:09.750103 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 prepare_pg_stats pg_stats(1 pgs tid 30440 v 602) v1 from osd.3 2013-04-03 10:37:09.750108 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:37:10.361243 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:32989/0) 2013-04-03 10:37:10.690207 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 encode_pending v394014 2013-04-03 10:37:10.691462 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 encode_pending v 29011 2013-04-03 10:37:10.741579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos 2013-04-03 10:37:10.741606 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos version 394014 summary v 394013 2013-04-03 10:37:10.741624 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos latest full 393798 2013-04-03 10:37:10.741644 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos applying incremental log 394014 2013-04-03 10:37:09.722408 mon.0 10.252.24.25:6789/0 72 : [INF] pgmap v29010: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 5460B/s wr, 0op/s 2013-04-03 10:37:10.741694 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 check_subs 2013-04-03 10:37:10.741726 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 create_pending v 394015 2013-04-03 10:37:10.741749 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:10.768503 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 update_from_paxos applying incremental 29011 2013-04-03 10:37:10.768522 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 v29011: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 716B/s wr, 0op/s 2013-04-03 10:37:10.768542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 send_pg_creates to 0 pgs 2013-04-03 10:37:10.768545 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 update_logger 2013-04-03 10:37:10.768554 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 create_pending v 29012 2013-04-03 10:37:10.768563 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:10.768581 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 check_osd_map already seen 602 >= 602 2013-04-03 10:37:10.768584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 update_logger 2013-04-03 10:37:10.768594 7fcdffcdb700 0 log [INF] : pgmap v29011: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 716B/s wr, 0op/s 2013-04-03 10:37:10.768650 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos 2013-04-03 10:37:10.768668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos version 394014 summary v 394014 2013-04-03 10:37:10.768687 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:10.768729 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos 2013-04-03 10:37:10.768744 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos version 394014 summary v 394014 2013-04-03 10:37:10.768753 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:10.768764 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:37:10.768782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:10.768793 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:37:10.768802 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 logging 2013-04-03 10:37:10.768595 mon.0 10.252.24.25:6789/0 73 : [INF] pgmap v29011: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 716B/s wr, 0op/s 2013-04-03 10:37:10.793348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos 2013-04-03 10:37:10.793366 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos version 394014 summary v 394014 2013-04-03 10:37:10.793381 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:11.581504 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140341 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:37:11.767366 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 encode_pending v394015 2013-04-03 10:37:11.802096 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos 2013-04-03 10:37:11.802124 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394014 2013-04-03 10:37:11.802139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos latest full 393798 2013-04-03 10:37:11.802156 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos applying incremental log 394015 2013-04-03 10:37:10.768595 mon.0 10.252.24.25:6789/0 73 : [INF] pgmap v29011: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 716B/s wr, 0op/s 2013-04-03 10:37:11.802204 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 check_subs 2013-04-03 10:37:11.802235 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 create_pending v 394016 2013-04-03 10:37:11.802257 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:11.802320 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos 2013-04-03 10:37:11.802336 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 2013-04-03 10:37:11.802357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:11.814072 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos 2013-04-03 10:37:11.814088 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 2013-04-03 10:37:11.814103 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:12.927507 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:12.927517 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a84340 from 10.252.24.88:6789/0 2013-04-03 10:37:12.927569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 56 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:37:13.334817 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 check_down_pgs 2013-04-03 10:37:13.334851 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 v29011: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 716B/s wr, 0op/s 2013-04-03 10:37:13.334879 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:37:13.334902 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:37:13.334940 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:37:13.334971 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos 2013-04-03 10:37:13.334990 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 2013-04-03 10:37:13.335004 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 log 2013-04-03 10:37:13.335021 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:13.335040 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:37:13.381165 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21712 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:37:14.803118 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos 2013-04-03 10:37:14.803139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 2013-04-03 10:37:14.803159 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:15.581644 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140342 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:37:16.927635 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:16.927646 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a84080 from 10.252.24.88:6789/0 2013-04-03 10:37:16.927698 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 57 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:37:17.381537 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21713 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:37:17.803117 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos 2013-04-03 10:37:17.803137 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 2013-04-03 10:37:17.803155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:18.335183 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 v29011: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 716B/s wr, 0op/s 2013-04-03 10:37:18.335229 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:37:18.335253 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:37:18.335296 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:37:18.335328 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos 2013-04-03 10:37:18.335347 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 2013-04-03 10:37:18.335355 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 log 2013-04-03 10:37:18.335372 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:18.335389 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:37:18.495886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:18.495895 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:37:18.495931 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 preprocess_query pg_stats(1 pgs tid 12 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:37:18.495953 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 prepare_update pg_stats(1 pgs tid 12 v 602) v1 from osd.2 10.252.24.30:6800/1508 2013-04-03 10:37:18.495959 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 prepare_pg_stats pg_stats(1 pgs tid 12 v 602) v1 from osd.2 2013-04-03 10:37:18.495966 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:37:18.546146 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 encode_pending v 29012 2013-04-03 10:37:18.579975 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 update_from_paxos applying incremental 29012 2013-04-03 10:37:18.579992 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 v29012: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3328B/s wr, 0op/s 2013-04-03 10:37:18.580009 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 send_pg_creates to 0 pgs 2013-04-03 10:37:18.580012 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 update_logger 2013-04-03 10:37:18.580022 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 create_pending v 29013 2013-04-03 10:37:18.580031 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 _updated_stats for osd.2 10.252.24.30:6800/1508 2013-04-03 10:37:18.580047 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 check_osd_map already seen 602 >= 602 2013-04-03 10:37:18.580049 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 update_logger 2013-04-03 10:37:18.580060 7fcdffcdb700 0 log [INF] : pgmap v29012: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3328B/s wr, 0op/s 2013-04-03 10:37:18.580121 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos 2013-04-03 10:37:18.580139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 2013-04-03 10:37:18.580155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:18.580200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos 2013-04-03 10:37:18.580215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 2013-04-03 10:37:18.580224 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:18.580236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:37:18.580258 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:18.580269 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:37:18.580279 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 logging 2013-04-03 10:37:18.580061 mon.0 10.252.24.25:6789/0 74 : [INF] pgmap v29012: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3328B/s wr, 0op/s 2013-04-03 10:37:18.605664 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos 2013-04-03 10:37:18.605681 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 2013-04-03 10:37:18.605696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:19.579748 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 encode_pending v394016 2013-04-03 10:37:19.581769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140343 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:37:19.606091 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos 2013-04-03 10:37:19.606118 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394015 2013-04-03 10:37:19.606133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos latest full 393798 2013-04-03 10:37:19.606152 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos applying incremental log 394016 2013-04-03 10:37:18.580061 mon.0 10.252.24.25:6789/0 74 : [INF] pgmap v29012: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3328B/s wr, 0op/s 2013-04-03 10:37:19.606200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 check_subs 2013-04-03 10:37:19.606231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 create_pending v 394017 2013-04-03 10:37:19.606254 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:19.606315 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos 2013-04-03 10:37:19.606331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394016 2013-04-03 10:37:19.606349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:19.619386 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos 2013-04-03 10:37:19.619403 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394016 2013-04-03 10:37:19.619418 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:20.927733 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:20.927744 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83b00 from 10.252.24.88:6789/0 2013-04-03 10:37:20.927795 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 58 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:37:21.381435 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21714 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:37:22.607431 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos 2013-04-03 10:37:22.607452 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394016 2013-04-03 10:37:22.607472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:23.335536 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 check_down_pgs 2013-04-03 10:37:23.335564 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 v29012: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3328B/s wr, 0op/s 2013-04-03 10:37:23.335592 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:37:23.335617 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:37:23.335655 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:37:23.335686 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos 2013-04-03 10:37:23.335705 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394016 2013-04-03 10:37:23.335715 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 log 2013-04-03 10:37:23.335732 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:23.335751 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:37:23.581875 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140344 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:37:24.574364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:24.574374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:37:24.574426 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 preprocess_query pg_stats(1 pgs tid 20 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:24.574450 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 prepare_update pg_stats(1 pgs tid 20 v 602) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:24.574456 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 prepare_pg_stats pg_stats(1 pgs tid 20 v 602) v1 from osd.1 2013-04-03 10:37:24.574463 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:37:24.624671 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 encode_pending v 29013 2013-04-03 10:37:24.672183 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 update_from_paxos applying incremental 29013 2013-04-03 10:37:24.672202 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 v29013: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2351B/s wr, 0op/s 2013-04-03 10:37:24.672221 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 send_pg_creates to 0 pgs 2013-04-03 10:37:24.672226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 update_logger 2013-04-03 10:37:24.672236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 create_pending v 29014 2013-04-03 10:37:24.672245 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:24.672262 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 check_osd_map already seen 602 >= 602 2013-04-03 10:37:24.672265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 update_logger 2013-04-03 10:37:24.672275 7fcdffcdb700 0 log [INF] : pgmap v29013: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2351B/s wr, 0op/s 2013-04-03 10:37:24.672341 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos 2013-04-03 10:37:24.672362 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394016 2013-04-03 10:37:24.672379 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:24.672423 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos 2013-04-03 10:37:24.672439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394016 2013-04-03 10:37:24.672447 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:24.672457 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:37:24.672479 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:24.672491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:37:24.672502 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 logging 2013-04-03 10:37:24.672283 mon.0 10.252.24.25:6789/0 75 : [INF] pgmap v29013: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2351B/s wr, 0op/s 2013-04-03 10:37:24.683328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos 2013-04-03 10:37:24.683345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394016 2013-04-03 10:37:24.683360 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:24.927900 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:24.927911 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83580 from 10.252.24.88:6789/0 2013-04-03 10:37:24.927962 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 59 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:37:25.381530 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21715 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:37:25.669814 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 encode_pending v394017 2013-04-03 10:37:25.704813 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos 2013-04-03 10:37:25.704838 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394016 2013-04-03 10:37:25.704853 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos latest full 393798 2013-04-03 10:37:25.704870 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos applying incremental log 394017 2013-04-03 10:37:24.672283 mon.0 10.252.24.25:6789/0 75 : [INF] pgmap v29013: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2351B/s wr, 0op/s 2013-04-03 10:37:25.704915 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 check_subs 2013-04-03 10:37:25.704947 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 create_pending v 394018 2013-04-03 10:37:25.704970 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:25.705033 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos 2013-04-03 10:37:25.705050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 2013-04-03 10:37:25.705068 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:25.720766 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos 2013-04-03 10:37:25.720783 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 2013-04-03 10:37:25.720798 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:27.582016 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140345 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:37:28.317771 7fcdfe8d5700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 ms_verify_authorizer 10.252.24.30:6789/0 mon protocol 2 2013-04-03 10:37:28.317905 7fcdfe8d5700 0 -- 10.252.24.25:6789/0 >> 10.252.24.30:6789/0 pipe(0x29b4500 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state connecting 2013-04-03 10:37:28.335889 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 check_down_pgs 2013-04-03 10:37:28.335918 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 v29013: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2351B/s wr, 0op/s 2013-04-03 10:37:28.335946 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:37:28.335971 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in 2013-04-03 10:37:28.336008 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:37:28.336039 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos 2013-04-03 10:37:28.336058 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 2013-04-03 10:37:28.336068 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 log 2013-04-03 10:37:28.336085 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:28.336104 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:37:28.339760 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:37:28.339767 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:37:28.339772 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start sync session already in progress for mon.1 10.252.24.30:6789/0 2013-04-03 10:37:28.339776 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start ignore stray message 2013-04-03 10:37:28.705567 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos 2013-04-03 10:37:28.705585 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 2013-04-03 10:37:28.705601 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:28.928027 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:28.928040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83000 from 10.252.24.88:6789/0 2013-04-03 10:37:28.928090 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 60 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:37:29.381653 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21716 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:37:29.962101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:29.962111 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20c5800 from 10.252.24.88:6789/0 2013-04-03 10:37:29.962161 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 preprocess_query osd_boot(osd.2 booted 0 v602) v3 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:29.962173 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 preprocess_boot from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:29.962178 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 prepare_update osd_boot(osd.2 booted 0 v602) v3 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:29.962181 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 prepare_boot from osd.2 10.252.24.30:6801/5656 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.2 eb9ff30e-cc75-49d3-8ef9-2f478e173371 e602 [101,602] lci=[601,602]) cluster_addr 10.252.24.30:6802/5656 hb_addr 10.252.24.30:6803/5656 2013-04-03 10:37:29.962191 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 prepare_boot was up, first marking down osd.2 10.252.24.30:6800/1508 2013-04-03 10:37:29.962195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 should_propose 2013-04-03 10:37:30.012342 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:37:30.012401 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 2013-04-03 10:37:30.012454 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 encode_pending e 603 2013-04-03 10:37:30.012458 7fce004dc700 2 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 osd.2 DOWN 2013-04-03 10:37:30.063015 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 update_from_paxos applying incremental 603 2013-04-03 10:37:30.063059 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 e603: 5 osds: 4 up, 5 in 2013-04-03 10:37:30.063975 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 adding osd.2 to down_pending_out map 2013-04-03 10:37:30.064031 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 check_osd_map applying osdmap e603 to pg_map 2013-04-03 10:37:30.064050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 register_new_pgs checking pg pools for osdmap epoch 603, last_pg_scan 341 2013-04-03 10:37:30.064055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 2013-04-03 10:37:30.064066 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:37:30.064069 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:37:30.064072 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 2013-04-03 10:37:30.064075 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 2013-04-03 10:37:30.064078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 2013-04-03 10:37:30.064083 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 2013-04-03 10:37:30.064086 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 2013-04-03 10:37:30.064088 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 2013-04-03 10:37:30.064099 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs 2013-04-03 10:37:30.064101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 check_down_pgs 2013-04-03 10:37:30.064103 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.d stale with acting [2,0] 2013-04-03 10:37:30.064108 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 8.3 stale with acting [2,3] 2013-04-03 10:37:30.064111 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.9 stale with acting [2,3] 2013-04-03 10:37:30.064114 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.a stale with acting [2,3] 2013-04-03 10:37:30.064116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.b stale with acting [2,3] 2013-04-03 10:37:30.064120 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 7.0 stale with acting [2,3] 2013-04-03 10:37:30.064122 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 6.1 stale with acting [2,3] 2013-04-03 10:37:30.064125 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 5.2 stale with acting [2,3] 2013-04-03 10:37:30.064128 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 4.3 stale with acting [2,3] 2013-04-03 10:37:30.064130 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 3.4 stale with acting [2,3] 2013-04-03 10:37:30.064133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.5 stale with acting [2,3] 2013-04-03 10:37:30.064136 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.6 stale with acting [2,3] 2013-04-03 10:37:30.064138 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.7 stale with acting [2,3] 2013-04-03 10:37:30.064141 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 5.0 stale with acting [2,0] 2013-04-03 10:37:30.064147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 4.1 stale with acting [2,0] 2013-04-03 10:37:30.064150 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.4 stale with acting [2,0] 2013-04-03 10:37:30.064152 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.5 stale with acting [2,0] 2013-04-03 10:37:30.064155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 7.4 stale with acting [2,3] 2013-04-03 10:37:30.064158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 6.5 stale with acting [2,3] 2013-04-03 10:37:30.064161 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 5.6 stale with acting [2,3] 2013-04-03 10:37:30.064166 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 4.7 stale with acting [2,3] 2013-04-03 10:37:30.064169 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.1 stale with acting [2,4] 2013-04-03 10:37:30.064172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.2 stale with acting [2,4] 2013-04-03 10:37:30.064174 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.3 stale with acting [2,4] 2013-04-03 10:37:30.064177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 3.0 stale with acting [2,4] 2013-04-03 10:37:30.064179 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 3.2 stale with acting [2,0] 2013-04-03 10:37:30.064182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.3 stale with acting [2,0] 2013-04-03 10:37:30.064185 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 7.7 stale with acting [2,0] 2013-04-03 10:37:30.064189 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.3c stale with acting [2,3] 2013-04-03 10:37:30.064191 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.3e stale with acting [2,3] 2013-04-03 10:37:30.064194 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.3d stale with acting [2,3] 2013-04-03 10:37:30.064197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.39 stale with acting [2,0] 2013-04-03 10:37:30.064200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.3a stale with acting [2,0] 2013-04-03 10:37:30.064205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.3b stale with acting [2,0] 2013-04-03 10:37:30.064208 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.38 stale with acting [2,3] 2013-04-03 10:37:30.064211 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.3a stale with acting [2,3] 2013-04-03 10:37:30.064213 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.39 stale with acting [2,3] 2013-04-03 10:37:30.064217 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.34 stale with acting [2,1] 2013-04-03 10:37:30.064220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.36 stale with acting [2,1] 2013-04-03 10:37:30.064222 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.35 stale with acting [2,1] 2013-04-03 10:37:30.064226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.29 stale with acting [2,0] 2013-04-03 10:37:30.064228 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.2a stale with acting [2,0] 2013-04-03 10:37:30.064231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.2b stale with acting [2,0] 2013-04-03 10:37:30.064234 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.28 stale with acting [2,0] 2013-04-03 10:37:30.064236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.2a stale with acting [2,0] 2013-04-03 10:37:30.064239 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.28 stale with acting [2,1] 2013-04-03 10:37:30.064241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.29 stale with acting [2,1] 2013-04-03 10:37:30.064244 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.29 stale with acting [2,0] 2013-04-03 10:37:30.064247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.27 stale with acting [2,1] 2013-04-03 10:37:30.064250 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.21 stale with acting [2,3] 2013-04-03 10:37:30.064252 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.22 stale with acting [2,3] 2013-04-03 10:37:30.064255 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.23 stale with acting [2,3] 2013-04-03 10:37:30.064261 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.20 stale with acting [2,1] 2013-04-03 10:37:30.064264 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.22 stale with acting [2,1] 2013-04-03 10:37:30.064267 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.21 stale with acting [2,1] 2013-04-03 10:37:30.064270 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.1c stale with acting [2,3] 2013-04-03 10:37:30.064275 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.1b stale with acting [2,3] 2013-04-03 10:37:30.064278 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.19 stale with acting [2,3] 2013-04-03 10:37:30.064281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.18 stale with acting [2,3] 2013-04-03 10:37:30.064284 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.18 stale with acting [2,3] 2013-04-03 10:37:30.064287 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.1a stale with acting [2,3] 2013-04-03 10:37:30.064290 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.17 stale with acting [2,3] 2013-04-03 10:37:30.064292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.17 stale with acting [2,3] 2013-04-03 10:37:30.064295 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.16 stale with acting [2,3] 2013-04-03 10:37:30.064299 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.f stale with acting [2,3] 2013-04-03 10:37:30.064301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.e stale with acting [2,3] 2013-04-03 10:37:30.064304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.d stale with acting [2,3] 2013-04-03 10:37:30.064307 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 8.7 stale with acting [2,3] 2013-04-03 10:37:30.064310 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.e stale with acting [2,0] 2013-04-03 10:37:30.064312 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.c stale with acting [2,0] 2013-04-03 10:37:30.064315 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 8.6 stale with acting [2,0] 2013-04-03 10:37:30.064370 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 encode_pending v 29014 2013-04-03 10:37:30.067442 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 send_pg_creates to 0 pgs 2013-04-03 10:37:30.067447 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 send_to_waiting 603 2013-04-03 10:37:30.067450 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 share_map_with_random_osd no up osd on our session map 2013-04-03 10:37:30.067452 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 update_logger 2013-04-03 10:37:30.067472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 create_pending e 604 2013-04-03 10:37:30.067475 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 remove_redundant_pg_temp 2013-04-03 10:37:30.067477 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 remove_down_pg_temp 2013-04-03 10:37:30.067514 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_query osd_boot(osd.2 booted 0 v602) v3 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:30.067523 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_boot from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:30.067526 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_update osd_boot(osd.2 booted 0 v602) v3 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:30.067530 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_boot from osd.2 10.252.24.30:6801/5656 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.2 eb9ff30e-cc75-49d3-8ef9-2f478e173371 e602 [101,602] lci=[601,602]) cluster_addr 10.252.24.30:6802/5656 hb_addr 10.252.24.30:6803/5656 2013-04-03 10:37:30.067540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 setting osd.2 uuid to eb9ff30e-cc75-49d3-8ef9-2f478e173371 2013-04-03 10:37:30.067544 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 old osd_info: up_from 601 up_thru 601 down_at 603 last_clean_interval [593,599) lost_at 95 2013-04-03 10:37:30.067546 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_boot osd.2 last_clean_interval [593,599) -> [601-602) 2013-04-03 10:37:30.067549 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 not laggy, new xi down_stamp 2013-04-03 10:37:30.012458 laggy_probability 0 laggy_interval 0 2013-04-03 10:37:30.067560 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 should_propose 2013-04-03 10:37:30.067572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 update_logger 2013-04-03 10:37:30.067579 7fcdffcdb700 0 log [INF] : osdmap e603: 5 osds: 4 up, 5 in 2013-04-03 10:37:30.067641 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos 2013-04-03 10:37:30.067658 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 2013-04-03 10:37:30.067668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:30.067679 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:37:30.067702 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:30.067715 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:37:30.068262 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 logging 2013-04-03 10:37:30.067580 mon.0 10.252.24.25:6789/0 76 : [INF] osdmap e603: 5 osds: 4 up, 5 in 2013-04-03 10:37:30.126323 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 update_from_paxos applying incremental 29014 2013-04-03 10:37:30.126915 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 v29014: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 355B/s wr, 0op/s 2013-04-03 10:37:30.126956 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 send_pg_creates to 0 pgs 2013-04-03 10:37:30.126959 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 update_logger 2013-04-03 10:37:30.126981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 create_pending v 29015 2013-04-03 10:37:30.126991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 check_osd_map already seen 603 >= 603 2013-04-03 10:37:30.126993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 update_logger 2013-04-03 10:37:30.127005 7fcdffcdb700 0 log [INF] : pgmap v29014: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 355B/s wr, 0op/s 2013-04-03 10:37:30.127063 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos 2013-04-03 10:37:30.127084 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 2013-04-03 10:37:30.127102 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:30.127142 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos 2013-04-03 10:37:30.127155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 2013-04-03 10:37:30.127163 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:30.127177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:37:30.127199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 nothing new 2013-04-03 10:37:30.227477 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos 2013-04-03 10:37:30.227497 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 2013-04-03 10:37:30.227514 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:30.227540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:30.227549 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x28a5700 from 10.252.24.88:6789/0 2013-04-03 10:37:30.227572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_query osd_alive(want up_thru 603 have 603) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:37:30.227580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_alive want up_thru 603 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:37:30.227585 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_update osd_alive(want up_thru 603 have 603) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:37:30.227589 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_alive want up_thru 603 have 603 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:37:30.227593 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 should_propose 2013-04-03 10:37:30.227820 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:30.227827 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x28a5540 from 10.252.24.88:6789/0 2013-04-03 10:37:30.227851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_query osd_alive(want up_thru 603 have 603) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:30.227858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_alive want up_thru 603 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:30.227864 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_update osd_alive(want up_thru 603 have 603) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:30.227867 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_alive want up_thru 603 have 603 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:30.227870 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 should_propose 2013-04-03 10:37:30.227888 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:30.227893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x28a5380 from 10.252.24.88:6789/0 2013-04-03 10:37:30.227914 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_query osd_alive(want up_thru 603 have 603) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:37:30.227919 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_alive want up_thru 603 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:37:30.227922 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_update osd_alive(want up_thru 603 have 603) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:37:30.227926 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_alive want up_thru 603 have 603 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:37:30.227929 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 should_propose 2013-04-03 10:37:30.393194 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acdc80 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33051/0) 2013-04-03 10:37:31.051165 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 encode_pending v394018 2013-04-03 10:37:31.052850 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 min_last_epoch_clean 593 2013-04-03 10:37:31.052907 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 min_last_epoch_clean 593 2013-04-03 10:37:31.052957 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 encode_pending e 604 2013-04-03 10:37:31.052962 7fce004dc700 2 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 osd.2 UP 10.252.24.30:6801/5656 2013-04-03 10:37:31.097718 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos 2013-04-03 10:37:31.097743 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos version 394018 summary v 394017 2013-04-03 10:37:31.097758 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos latest full 393798 2013-04-03 10:37:31.097775 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos applying incremental log 394018 2013-04-03 10:37:30.067580 mon.0 10.252.24.25:6789/0 76 : [INF] osdmap e603: 5 osds: 4 up, 5 in 2013-04-03 10:37:31.097820 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 check_subs 2013-04-03 10:37:31.097851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 create_pending v 394019 2013-04-03 10:37:31.097876 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:31.123532 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 update_from_paxos applying incremental 604 2013-04-03 10:37:31.123582 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 e604: 5 osds: 5 up, 5 in 2013-04-03 10:37:31.124833 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 check_osd_map applying osdmap e604 to pg_map 2013-04-03 10:37:31.124863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 register_new_pgs checking pg pools for osdmap epoch 604, last_pg_scan 341 2013-04-03 10:37:31.124867 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 2013-04-03 10:37:31.124872 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:37:31.124875 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:37:31.124878 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 2013-04-03 10:37:31.124881 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 2013-04-03 10:37:31.124883 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 2013-04-03 10:37:31.124886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 2013-04-03 10:37:31.124889 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 2013-04-03 10:37:31.124892 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 2013-04-03 10:37:31.124903 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs 2013-04-03 10:37:31.124905 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 check_down_pgs 2013-04-03 10:37:31.124961 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 encode_pending v 29015 2013-04-03 10:37:31.125210 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 send_pg_creates to 0 pgs 2013-04-03 10:37:31.125215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_to_waiting 604 2013-04-03 10:37:31.125218 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 share_map_with_random_osd no up osd on our session map 2013-04-03 10:37:31.125225 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 update_logger 2013-04-03 10:37:31.125248 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 create_pending e 605 2013-04-03 10:37:31.125251 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 remove_redundant_pg_temp 2013-04-03 10:37:31.125253 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 remove_down_pg_temp 2013-04-03 10:37:31.125271 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 _booted osd.2 10.252.24.30:6801/5656 w 0 from 602 2013-04-03 10:37:31.125288 7fcdffcdb700 0 log [INF] : osd.2 10.252.24.30:6801/5656 boot 2013-04-03 10:37:31.125319 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_latest to osd.2 10.252.24.30:6801/5656 start 603 2013-04-03 10:37:31.125326 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_incremental [603..604] to osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:31.125338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 build_incremental [603..604] 2013-04-03 10:37:31.125384 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 _reply_map 603 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:37:31.125402 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_latest to osd.4 10.252.24.32:6800/13473 start 603 2013-04-03 10:37:31.125408 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_incremental [603..604] to osd.4 10.252.24.32:6800/13473 2013-04-03 10:37:31.125419 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 build_incremental [603..604] 2013-04-03 10:37:31.125461 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 _reply_map 603 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:31.125484 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_latest to osd.3 10.252.24.88:6801/17538 start 603 2013-04-03 10:37:31.125491 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_incremental [603..604] to osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:31.125502 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 build_incremental [603..604] 2013-04-03 10:37:31.125542 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 _reply_map 603 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:37:31.125560 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_latest to osd.0 10.252.58.31:6802/2682 start 603 2013-04-03 10:37:31.125564 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_incremental [603..604] to osd.0 10.252.58.31:6802/2682 2013-04-03 10:37:31.125575 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 build_incremental [603..604] 2013-04-03 10:37:31.125615 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 update_logger 2013-04-03 10:37:31.125621 7fcdffcdb700 0 log [INF] : osdmap e604: 5 osds: 5 up, 5 in 2013-04-03 10:37:31.125675 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos 2013-04-03 10:37:31.125693 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos version 394018 summary v 394018 2013-04-03 10:37:31.125702 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:31.125713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:37:31.125739 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:31.125751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:37:31.125762 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 logging 2013-04-03 10:37:30.127006 mon.0 10.252.24.25:6789/0 77 : [INF] pgmap v29014: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 355B/s wr, 0op/s 2013-04-03 10:37:31.125781 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 logging 2013-04-03 10:37:31.125289 mon.0 10.252.24.25:6789/0 78 : [INF] osd.2 10.252.24.30:6801/5656 boot 2013-04-03 10:37:31.125843 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos 2013-04-03 10:37:31.125858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos version 394018 summary v 394018 2013-04-03 10:37:31.125866 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:31.125876 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:37:31.125892 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 nothing new 2013-04-03 10:37:31.188143 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 update_from_paxos applying incremental 29015 2013-04-03 10:37:31.188155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 v29015: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:31.188175 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 send_pg_creates to 0 pgs 2013-04-03 10:37:31.188178 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 update_logger 2013-04-03 10:37:31.188188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 create_pending v 29016 2013-04-03 10:37:31.188197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 check_osd_map already seen 604 >= 604 2013-04-03 10:37:31.188200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 update_logger 2013-04-03 10:37:31.188210 7fcdffcdb700 0 log [INF] : pgmap v29015: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:31.188272 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos 2013-04-03 10:37:31.188292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos version 394018 summary v 394018 2013-04-03 10:37:31.188311 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:31.188353 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos 2013-04-03 10:37:31.188368 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos version 394018 summary v 394018 2013-04-03 10:37:31.188378 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:31.188389 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:37:31.188420 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:31.188432 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:37:31.188443 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 logging 2013-04-03 10:37:31.125289 mon.0 10.252.24.25:6789/0 78 : [INF] osd.2 10.252.24.30:6801/5656 boot 2013-04-03 10:37:31.188462 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 logging 2013-04-03 10:37:31.125622 mon.0 10.252.24.25:6789/0 79 : [INF] osdmap e604: 5 osds: 5 up, 5 in 2013-04-03 10:37:31.241211 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos 2013-04-03 10:37:31.241228 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos version 394018 summary v 394018 2013-04-03 10:37:31.241243 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:31.241488 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:31.241497 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x28a51c0 from 10.252.24.88:6789/0 2013-04-03 10:37:31.241523 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_query osd_alive(want up_thru 604 have 604) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:31.241530 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_alive want up_thru 604 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:31.241534 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_update osd_alive(want up_thru 604 have 604) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:31.241538 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_alive want up_thru 604 have 604 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:31.241542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 should_propose 2013-04-03 10:37:31.241560 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:31.241565 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2294e00 from 10.252.24.88:6789/0 2013-04-03 10:37:31.241587 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_query osd_alive(want up_thru 604 have 604) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:31.241592 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_alive want up_thru 604 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:31.241596 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_update osd_alive(want up_thru 604 have 604) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:31.241599 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_alive want up_thru 604 have 604 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:31.241602 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 should_propose 2013-04-03 10:37:31.244240 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:31.244248 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2294540 from 10.252.24.88:6789/0 2013-04-03 10:37:31.244279 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_query osd_alive(want up_thru 604 have 604) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:31.244287 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_alive want up_thru 604 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:31.244291 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_update osd_alive(want up_thru 604 have 604) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:31.244294 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_alive want up_thru 604 have 604 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:31.244298 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 should_propose 2013-04-03 10:37:31.249686 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:31.249694 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2294a80 from 10.252.24.88:6789/0 2013-04-03 10:37:31.249726 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_query osd_alive(want up_thru 604 have 604) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:37:31.249733 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_alive want up_thru 604 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:37:31.249736 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_update osd_alive(want up_thru 604 have 604) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:37:31.249741 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_alive want up_thru 604 have 604 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:37:31.249744 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 should_propose 2013-04-03 10:37:31.582148 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140346 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:37:32.123385 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 encode_pending v394019 2013-04-03 10:37:32.172866 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos 2013-04-03 10:37:32.172902 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos version 394019 summary v 394018 2013-04-03 10:37:32.172917 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos latest full 393798 2013-04-03 10:37:32.172936 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos applying incremental log 394019 2013-04-03 10:37:30.127006 mon.0 10.252.24.25:6789/0 77 : [INF] pgmap v29014: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 355B/s wr, 0op/s 2013-04-03 10:37:32.172965 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos applying incremental log 394019 2013-04-03 10:37:31.125289 mon.0 10.252.24.25:6789/0 78 : [INF] osd.2 10.252.24.30:6801/5656 boot 2013-04-03 10:37:32.172985 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos applying incremental log 394019 2013-04-03 10:37:31.125622 mon.0 10.252.24.25:6789/0 79 : [INF] osdmap e604: 5 osds: 5 up, 5 in 2013-04-03 10:37:32.173021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 check_subs 2013-04-03 10:37:32.173055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 create_pending v 394020 2013-04-03 10:37:32.173078 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:32.173099 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:32.173163 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos 2013-04-03 10:37:32.173179 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos version 394019 summary v 394019 2013-04-03 10:37:32.173197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:32.187831 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 min_last_epoch_clean 593 2013-04-03 10:37:32.187892 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 min_last_epoch_clean 593 2013-04-03 10:37:32.188585 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 encode_pending e 605 2013-04-03 10:37:32.211935 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 update_from_paxos applying incremental 605 2013-04-03 10:37:32.211981 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:37:32.213128 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 check_osd_map applying osdmap e605 to pg_map 2013-04-03 10:37:32.213151 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 register_new_pgs checking pg pools for osdmap epoch 605, last_pg_scan 341 2013-04-03 10:37:32.213158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 2013-04-03 10:37:32.213163 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:37:32.213165 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:37:32.213168 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 2013-04-03 10:37:32.213171 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 2013-04-03 10:37:32.213174 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 2013-04-03 10:37:32.213177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 2013-04-03 10:37:32.213180 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 2013-04-03 10:37:32.213182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 2013-04-03 10:37:32.213192 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs 2013-04-03 10:37:32.213195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 check_down_pgs 2013-04-03 10:37:32.213256 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 encode_pending v 29016 2013-04-03 10:37:32.213535 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 send_pg_creates to 0 pgs 2013-04-03 10:37:32.213545 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_to_waiting 605 2013-04-03 10:37:32.213549 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 share_map_with_random_osd no up osd on our session map 2013-04-03 10:37:32.213551 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 update_logger 2013-04-03 10:37:32.213576 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 create_pending e 606 2013-04-03 10:37:32.213579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 remove_redundant_pg_temp 2013-04-03 10:37:32.213581 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 remove_down_pg_temp 2013-04-03 10:37:32.213600 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 _reply_map 604 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:32.213618 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_latest to osd.2 10.252.24.30:6801/5656 start 604 2013-04-03 10:37:32.213622 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_incremental [604..605] to osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:32.213633 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 build_incremental [604..605] 2013-04-03 10:37:32.213677 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 _reply_map 604 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:32.213696 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_latest to osd.3 10.252.24.88:6801/17538 start 604 2013-04-03 10:37:32.213700 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_incremental [604..605] to osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:32.213710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 build_incremental [604..605] 2013-04-03 10:37:32.213750 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 _reply_map 604 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:32.213766 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_latest to osd.1 10.252.24.25:6805/3572 start 604 2013-04-03 10:37:32.213770 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_incremental [604..605] to osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:32.213780 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 build_incremental [604..605] 2013-04-03 10:37:32.213831 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 _reply_map 604 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:37:32.213859 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_latest to osd.0 10.252.58.31:6802/2682 start 604 2013-04-03 10:37:32.213864 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_incremental [604..605] to osd.0 10.252.58.31:6802/2682 2013-04-03 10:37:32.213874 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 build_incremental [604..605] 2013-04-03 10:37:32.213918 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 update_logger 2013-04-03 10:37:32.213926 7fcdffcdb700 0 log [INF] : osdmap e605: 5 osds: 5 up, 5 in 2013-04-03 10:37:32.213988 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos 2013-04-03 10:37:32.214002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos version 394019 summary v 394019 2013-04-03 10:37:32.214010 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:32.214020 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:37:32.214047 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:32.214059 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:37:32.214070 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 logging 2013-04-03 10:37:31.188212 mon.0 10.252.24.25:6789/0 80 : [INF] pgmap v29015: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:32.214091 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 logging 2013-04-03 10:37:32.213927 mon.0 10.252.24.25:6789/0 81 : [INF] osdmap e605: 5 osds: 5 up, 5 in 2013-04-03 10:37:32.274625 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 update_from_paxos applying incremental 29016 2013-04-03 10:37:32.274634 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 v29016: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:32.274652 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 send_pg_creates to 0 pgs 2013-04-03 10:37:32.274655 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 update_logger 2013-04-03 10:37:32.274665 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 create_pending v 29017 2013-04-03 10:37:32.274674 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 check_osd_map already seen 605 >= 605 2013-04-03 10:37:32.274676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 update_logger 2013-04-03 10:37:32.274686 7fcdffcdb700 0 log [INF] : pgmap v29016: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:32.274740 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos 2013-04-03 10:37:32.274759 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos version 394019 summary v 394019 2013-04-03 10:37:32.274779 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:32.274822 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos 2013-04-03 10:37:32.274837 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos version 394019 summary v 394019 2013-04-03 10:37:32.274847 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:32.274858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:37:32.274879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 nothing new 2013-04-03 10:37:32.301487 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos 2013-04-03 10:37:32.301504 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos version 394019 summary v 394019 2013-04-03 10:37:32.301519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:32.928127 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:32.928138 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20e4340 from 10.252.24.88:6789/0 2013-04-03 10:37:32.928182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 61 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:37:33.211759 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 encode_pending v394020 2013-04-03 10:37:33.263888 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos 2013-04-03 10:37:33.263926 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos version 394020 summary v 394019 2013-04-03 10:37:33.263940 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos latest full 393798 2013-04-03 10:37:33.263957 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos applying incremental log 394020 2013-04-03 10:37:31.188212 mon.0 10.252.24.25:6789/0 80 : [INF] pgmap v29015: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:33.263986 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos applying incremental log 394020 2013-04-03 10:37:32.213927 mon.0 10.252.24.25:6789/0 81 : [INF] osdmap e605: 5 osds: 5 up, 5 in 2013-04-03 10:37:33.264023 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 check_subs 2013-04-03 10:37:33.264055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 create_pending v 394021 2013-04-03 10:37:33.264077 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:33.264142 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos 2013-04-03 10:37:33.264158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos version 394020 summary v 394020 2013-04-03 10:37:33.264461 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:33.311063 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos 2013-04-03 10:37:33.311081 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos version 394020 summary v 394020 2013-04-03 10:37:33.311098 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:33.313840 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:33.313849 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26d80 from 10.252.24.88:6789/0 2013-04-03 10:37:33.313880 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 preprocess_query pg_stats(71 pgs tid 1 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:33.313888 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 prepare_update pg_stats(71 pgs tid 1 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:33.313893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 prepare_pg_stats pg_stats(71 pgs tid 1 v 605) v1 from osd.2 2013-04-03 10:37:33.313899 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,3]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:37:33.336219 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 check_down_pgs 2013-04-03 10:37:33.336255 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 v29016: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:33.336284 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:37:33.336306 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:37:33.336344 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:37:33.336374 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos 2013-04-03 10:37:33.336392 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos version 394020 summary v 394020 2013-04-03 10:37:33.336400 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 log 2013-04-03 10:37:33.336415 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:33.336432 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:37:33.381680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21717 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:37:34.237113 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 encode_pending v 29017 2013-04-03 10:37:34.582152 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 update_from_paxos applying incremental 29017 2013-04-03 10:37:34.582336 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 v29017: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 23690B/s wr, 0op/s 2013-04-03 10:37:34.582367 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 send_pg_creates to 0 pgs 2013-04-03 10:37:34.582371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 update_logger 2013-04-03 10:37:34.582388 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 create_pending v 29018 2013-04-03 10:37:34.582402 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:34.582432 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 check_osd_map already seen 605 >= 605 2013-04-03 10:37:34.582436 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 update_logger 2013-04-03 10:37:34.582447 7fcdffcdb700 0 log [INF] : pgmap v29017: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 23690B/s wr, 0op/s 2013-04-03 10:37:34.582516 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos 2013-04-03 10:37:34.582536 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos version 394020 summary v 394020 2013-04-03 10:37:34.582555 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:34.582601 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos 2013-04-03 10:37:34.582616 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos version 394020 summary v 394020 2013-04-03 10:37:34.582626 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:34.582637 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:37:34.582666 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:34.582680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:37:34.582691 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 logging 2013-04-03 10:37:32.274687 mon.0 10.252.24.25:6789/0 82 : [INF] pgmap v29016: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:34.582711 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 logging 2013-04-03 10:37:34.582448 mon.0 10.252.24.25:6789/0 83 : [INF] pgmap v29017: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 23690B/s wr, 0op/s 2013-04-03 10:37:34.627562 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos 2013-04-03 10:37:34.627580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos version 394020 summary v 394020 2013-04-03 10:37:34.627596 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:35.440619 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:35.440630 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:37:35.440681 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 preprocess_query pg_stats(0 pgs tid 59 v 605) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:37:35.440689 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 prepare_update pg_stats(0 pgs tid 59 v 605) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:37:35.440693 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 prepare_pg_stats pg_stats(0 pgs tid 59 v 605) v1 from osd.4 2013-04-03 10:37:35.440699 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 got osd.4 osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[])) 2013-04-03 10:37:35.580959 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 encode_pending v394021 2013-04-03 10:37:35.582450 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 encode_pending v 29018 2013-04-03 10:37:35.582539 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140347 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:37:35.753040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos 2013-04-03 10:37:35.753067 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos version 394021 summary v 394020 2013-04-03 10:37:35.753082 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos latest full 393798 2013-04-03 10:37:35.753100 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos applying incremental log 394021 2013-04-03 10:37:32.274687 mon.0 10.252.24.25:6789/0 82 : [INF] pgmap v29016: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:35.753141 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos applying incremental log 394021 2013-04-03 10:37:34.582448 mon.0 10.252.24.25:6789/0 83 : [INF] pgmap v29017: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 23690B/s wr, 0op/s 2013-04-03 10:37:35.753183 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 check_subs 2013-04-03 10:37:35.753214 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 create_pending v 394022 2013-04-03 10:37:35.753237 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:35.837247 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 update_from_paxos applying incremental 29018 2013-04-03 10:37:35.837260 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 v29018: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 21882B/s wr, 0op/s 2013-04-03 10:37:35.837643 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 send_pg_creates to 0 pgs 2013-04-03 10:37:35.837648 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 update_logger 2013-04-03 10:37:35.837659 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 create_pending v 29019 2013-04-03 10:37:35.837668 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 _updated_stats for osd.4 10.252.24.32:6800/13473 2013-04-03 10:37:35.837685 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 check_osd_map already seen 605 >= 605 2013-04-03 10:37:35.837688 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 update_logger 2013-04-03 10:37:35.837700 7fcdffcdb700 0 log [INF] : pgmap v29018: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 21882B/s wr, 0op/s 2013-04-03 10:37:35.837763 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos 2013-04-03 10:37:35.837782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos version 394021 summary v 394021 2013-04-03 10:37:35.837804 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:35.837863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos 2013-04-03 10:37:35.837879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos version 394021 summary v 394021 2013-04-03 10:37:35.837888 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:35.837899 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:37:35.837922 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:35.837933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:37:35.837943 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 logging 2013-04-03 10:37:35.837701 mon.0 10.252.24.25:6789/0 84 : [INF] pgmap v29018: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 21882B/s wr, 0op/s 2013-04-03 10:37:35.850439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos 2013-04-03 10:37:35.850456 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos version 394021 summary v 394021 2013-04-03 10:37:35.850473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:36.579275 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:36.579285 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26d80 from 10.252.24.88:6789/0 2013-04-03 10:37:36.579337 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 preprocess_query pg_stats(45 pgs tid 21 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:36.579346 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 prepare_update pg_stats(45 pgs tid 21 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:36.579350 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 prepare_pg_stats pg_stats(45 pgs tid 21 v 605) v1 from osd.1 2013-04-03 10:37:36.579357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) 2013-04-03 10:37:36.744832 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:36.744841 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:37:36.744870 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 preprocess_query pg_stats(73 pgs tid 30441 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:36.744878 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 prepare_update pg_stats(73 pgs tid 30441 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:36.744882 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 prepare_pg_stats pg_stats(73 pgs tid 30441 v 605) v1 from osd.3 2013-04-03 10:37:36.744886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 got osd.3 osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:37:36.809660 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:36.809668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:37:36.809697 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 preprocess_query pg_stats(51 pgs tid 408 v 605) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:37:36.809704 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 prepare_update pg_stats(51 pgs tid 408 v 605) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:37:36.809708 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 prepare_pg_stats pg_stats(51 pgs tid 408 v 605) v1 from osd.0 2013-04-03 10:37:36.809713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 got osd.0 osd_stat(1139 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1139 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:37:36.836941 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 encode_pending v394022 2013-04-03 10:37:36.838589 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 encode_pending v 29019 2013-04-03 10:37:36.876205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos 2013-04-03 10:37:36.876243 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos version 394022 summary v 394021 2013-04-03 10:37:36.876258 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos latest full 393798 2013-04-03 10:37:36.876280 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos applying incremental log 394022 2013-04-03 10:37:35.837701 mon.0 10.252.24.25:6789/0 84 : [INF] pgmap v29018: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 21882B/s wr, 0op/s 2013-04-03 10:37:36.876328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 check_subs 2013-04-03 10:37:36.876361 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 create_pending v 394023 2013-04-03 10:37:36.876384 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:36.929081 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 update_from_paxos applying incremental 29019 2013-04-03 10:37:36.929496 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 v29019: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:36.929537 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 send_pg_creates to 0 pgs 2013-04-03 10:37:36.929544 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 update_logger 2013-04-03 10:37:36.929587 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 create_pending v 29020 2013-04-03 10:37:36.929601 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:36.929628 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:36.929688 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:37:36.929732 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 check_osd_map already seen 605 >= 605 2013-04-03 10:37:36.929741 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 update_logger 2013-04-03 10:37:36.929754 7fcdffcdb700 0 log [INF] : pgmap v29019: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:36.929835 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos 2013-04-03 10:37:36.929857 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos version 394022 summary v 394022 2013-04-03 10:37:36.929876 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:36.929917 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos 2013-04-03 10:37:36.929933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos version 394022 summary v 394022 2013-04-03 10:37:36.929944 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:36.929955 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:37:36.929978 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:36.929990 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:37:36.930000 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 logging 2013-04-03 10:37:36.929756 mon.0 10.252.24.25:6789/0 85 : [INF] pgmap v29019: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:36.960774 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos 2013-04-03 10:37:36.960793 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos version 394022 summary v 394022 2013-04-03 10:37:36.960809 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:36.960835 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:36.960844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29c4b00 from 10.252.24.88:6789/0 2013-04-03 10:37:36.960868 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 62 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:37:37.381838 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21718 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:37:37.927768 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 encode_pending v394023 2013-04-03 10:37:37.953542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos 2013-04-03 10:37:37.953568 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos version 394023 summary v 394022 2013-04-03 10:37:37.953582 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos latest full 393798 2013-04-03 10:37:37.953599 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos applying incremental log 394023 2013-04-03 10:37:36.929756 mon.0 10.252.24.25:6789/0 85 : [INF] pgmap v29019: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:37.953648 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 check_subs 2013-04-03 10:37:37.953679 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 create_pending v 394024 2013-04-03 10:37:37.953702 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:37.954044 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos 2013-04-03 10:37:37.954066 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos version 394023 summary v 394023 2013-04-03 10:37:37.954086 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:37.978560 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos 2013-04-03 10:37:37.978577 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos version 394023 summary v 394023 2013-04-03 10:37:37.978592 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:38.217787 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:38.217798 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:37:38.217848 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 preprocess_query pg_stats(65 pgs tid 2 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:38.217856 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 prepare_update pg_stats(65 pgs tid 2 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:38.217860 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 prepare_pg_stats pg_stats(65 pgs tid 2 v 605) v1 from osd.2 2013-04-03 10:37:38.217867 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,3]/[])) 2013-04-03 10:37:38.336554 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 check_down_pgs 2013-04-03 10:37:38.336585 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 v29019: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:38.336613 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:37:38.336638 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:37:38.336675 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:37:38.336706 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos 2013-04-03 10:37:38.336726 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos version 394023 summary v 394023 2013-04-03 10:37:38.336735 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 log 2013-04-03 10:37:38.336752 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:38.336771 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:37:38.952746 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 encode_pending v 29020 2013-04-03 10:37:38.988081 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 update_from_paxos applying incremental 29020 2013-04-03 10:37:38.988249 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29020 v29020: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:38.988278 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29020 send_pg_creates to 0 pgs 2013-04-03 10:37:38.988282 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29020 update_logger 2013-04-03 10:37:38.988299 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29020 create_pending v 29021 2013-04-03 10:37:38.988309 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29020 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:38.988343 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29020 check_osd_map already seen 605 >= 605 2013-04-03 10:37:38.988347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29020 update_logger 2013-04-03 10:37:38.988362 7fcdffcdb700 0 log [INF] : pgmap v29020: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:38.988425 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos 2013-04-03 10:37:38.988446 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos version 394023 summary v 394023 2013-04-03 10:37:38.988465 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:38.988508 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos 2013-04-03 10:37:38.988523 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos version 394023 summary v 394023 2013-04-03 10:37:38.988533 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:38.988544 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:37:38.988567 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:38.988579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:37:38.988589 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 logging 2013-04-03 10:37:38.988363 mon.0 10.252.24.25:6789/0 86 : [INF] pgmap v29020: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:39.021405 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos 2013-04-03 10:37:39.021422 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos version 394023 summary v 394023 2013-04-03 10:37:39.021442 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:37:39.193215 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) 2013-04-03 10:37:39.193259 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader 2013-04-03 10:37:39.582389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140348 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:37:39.987141 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394023 encode_pending v394024 2013-04-03 10:37:40.015002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos 2013-04-03 10:37:40.015026 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394023 2013-04-03 10:37:40.015040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos latest full 393798 2013-04-03 10:37:40.015056 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394024 update_from_paxos applying incremental log 394024 2013-04-03 10:37:38.988363 mon.0 10.252.24.25:6789/0 86 : [INF] pgmap v29020: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:37:40.015102 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 check_subs 2013-04-03 10:37:40.015133 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 create_pending v 394025 2013-04-03 10:37:40.015155 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394024 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:40.015218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos 2013-04-03 10:37:40.015234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394024 2013-04-03 10:37:40.015251 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:40.040297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos 2013-04-03 10:37:40.040314 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394024 2013-04-03 10:37:40.040328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:40.928382 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:40.928391 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c5600 from 10.252.24.88:6789/0 2013-04-03 10:37:40.928443 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 63 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:37:41.381953 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21719 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:37:43.015903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos 2013-04-03 10:37:43.015923 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394024 2013-04-03 10:37:43.015940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:43.213833 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:43.213843 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:37:43.213894 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29020 preprocess_query pg_stats(1 pgs tid 3 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:43.213901 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29020 prepare_update pg_stats(1 pgs tid 3 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:43.213905 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29020 prepare_pg_stats pg_stats(1 pgs tid 3 v 605) v1 from osd.2 2013-04-03 10:37:43.213911 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29020 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:37:43.264290 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29020 encode_pending v 29021 2013-04-03 10:37:43.336890 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:37:43.336933 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:37:43.336952 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos 2013-04-03 10:37:43.336971 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394024 2013-04-03 10:37:43.336980 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394024 log 2013-04-03 10:37:43.336997 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:43.337015 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:37:43.389660 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29020 update_from_paxos applying incremental 29021 2013-04-03 10:37:43.389675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 v29021: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2549B/s wr, 0op/s 2013-04-03 10:37:43.389693 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 send_pg_creates to 0 pgs 2013-04-03 10:37:43.389695 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 update_logger 2013-04-03 10:37:43.389705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 create_pending v 29022 2013-04-03 10:37:43.389713 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29021 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:43.389729 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 check_osd_map already seen 605 >= 605 2013-04-03 10:37:43.389732 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 update_logger 2013-04-03 10:37:43.389742 7fcdffcdb700 0 log [INF] : pgmap v29021: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2549B/s wr, 0op/s 2013-04-03 10:37:43.389802 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos 2013-04-03 10:37:43.389827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394024 2013-04-03 10:37:43.389845 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:43.389887 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos 2013-04-03 10:37:43.389902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394024 2013-04-03 10:37:43.389909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:43.389919 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:37:43.389941 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:43.389953 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:37:43.389964 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 logging 2013-04-03 10:37:43.389748 mon.0 10.252.24.25:6789/0 87 : [INF] pgmap v29021: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2549B/s wr, 0op/s 2013-04-03 10:37:43.402474 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos 2013-04-03 10:37:43.402489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394024 2013-04-03 10:37:43.402503 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:43.582476 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140349 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:37:44.389462 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394024 encode_pending v394025 2013-04-03 10:37:44.417394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos 2013-04-03 10:37:44.417420 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394024 2013-04-03 10:37:44.417434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos latest full 393798 2013-04-03 10:37:44.417451 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394025 update_from_paxos applying incremental log 394025 2013-04-03 10:37:43.389748 mon.0 10.252.24.25:6789/0 87 : [INF] pgmap v29021: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2549B/s wr, 0op/s 2013-04-03 10:37:44.417496 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 check_subs 2013-04-03 10:37:44.417527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 create_pending v 394026 2013-04-03 10:37:44.417681 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394025 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:44.417751 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos 2013-04-03 10:37:44.417767 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 2013-04-03 10:37:44.417783 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:44.430297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos 2013-04-03 10:37:44.430314 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 2013-04-03 10:37:44.430328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:44.928594 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:44.928604 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2145000 from 10.252.24.88:6789/0 2013-04-03 10:37:44.928655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 64 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:37:45.382048 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21720 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:37:47.418253 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos 2013-04-03 10:37:47.418272 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 2013-04-03 10:37:47.418289 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:47.582596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140350 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:37:48.337162 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29021 check_down_pgs 2013-04-03 10:37:48.337192 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29021 v29021: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2549B/s wr, 0op/s 2013-04-03 10:37:48.337219 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:37:48.337248 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:37:48.337285 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:37:48.337324 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos 2013-04-03 10:37:48.337344 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 2013-04-03 10:37:48.337353 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394025 log 2013-04-03 10:37:48.337370 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:48.337389 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:37:48.928667 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:48.928676 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x295d580 from 10.252.24.88:6789/0 2013-04-03 10:37:48.928714 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 65 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:37:49.382179 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21721 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:37:50.418379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos 2013-04-03 10:37:50.418399 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 2013-04-03 10:37:50.418415 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:51.577473 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:51.577483 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:37:51.577533 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 preprocess_query pg_stats(1 pgs tid 22 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:51.577541 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 prepare_update pg_stats(1 pgs tid 22 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:51.577545 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 prepare_pg_stats pg_stats(1 pgs tid 22 v 605) v1 from osd.1 2013-04-03 10:37:51.577553 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:37:51.582708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140351 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:37:51.627736 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29021 encode_pending v 29022 2013-04-03 10:37:51.657755 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29021 update_from_paxos applying incremental 29022 2013-04-03 10:37:51.657770 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 v29022: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1615B/s wr, 0op/s 2013-04-03 10:37:51.657787 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 send_pg_creates to 0 pgs 2013-04-03 10:37:51.657790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 update_logger 2013-04-03 10:37:51.657799 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 create_pending v 29023 2013-04-03 10:37:51.657808 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29022 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:37:51.657831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 check_osd_map already seen 605 >= 605 2013-04-03 10:37:51.657833 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 update_logger 2013-04-03 10:37:51.657844 7fcdffcdb700 0 log [INF] : pgmap v29022: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1615B/s wr, 0op/s 2013-04-03 10:37:51.657906 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos 2013-04-03 10:37:51.657925 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 2013-04-03 10:37:51.657941 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:51.657983 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos 2013-04-03 10:37:51.657998 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 2013-04-03 10:37:51.658005 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:51.658015 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:37:51.658037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:51.658049 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:37:51.658058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 logging 2013-04-03 10:37:51.657845 mon.0 10.252.24.25:6789/0 88 : [INF] pgmap v29022: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1615B/s wr, 0op/s 2013-04-03 10:37:51.669044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos 2013-04-03 10:37:51.669059 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 2013-04-03 10:37:51.669074 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:51.743465 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:51.743474 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:37:51.743499 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 preprocess_query pg_stats(1 pgs tid 30442 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:51.743504 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 prepare_update pg_stats(1 pgs tid 30442 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:51.743507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 prepare_pg_stats pg_stats(1 pgs tid 30442 v 605) v1 from osd.3 2013-04-03 10:37:51.743511 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 got osd.3 osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:37:52.657222 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29022 encode_pending v 29023 2013-04-03 10:37:52.658518 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394025 encode_pending v394026 2013-04-03 10:37:52.685299 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29022 update_from_paxos applying incremental 29023 2013-04-03 10:37:52.685319 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 v29023: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:37:52.685338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 send_pg_creates to 0 pgs 2013-04-03 10:37:52.685341 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 update_logger 2013-04-03 10:37:52.685350 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 create_pending v 29024 2013-04-03 10:37:52.685359 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29023 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:37:52.685376 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 check_osd_map already seen 605 >= 605 2013-04-03 10:37:52.685378 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 update_logger 2013-04-03 10:37:52.685388 7fcdffcdb700 0 log [INF] : pgmap v29023: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:37:52.710354 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos 2013-04-03 10:37:52.710378 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394025 2013-04-03 10:37:52.710392 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos latest full 393798 2013-04-03 10:37:52.710409 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394026 update_from_paxos applying incremental log 394026 2013-04-03 10:37:51.657845 mon.0 10.252.24.25:6789/0 88 : [INF] pgmap v29022: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1615B/s wr, 0op/s 2013-04-03 10:37:52.710453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 check_subs 2013-04-03 10:37:52.710483 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 create_pending v 394027 2013-04-03 10:37:52.710505 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394026 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:52.710537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos 2013-04-03 10:37:52.710553 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 2013-04-03 10:37:52.710562 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:52.710573 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:37:52.710589 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 nothing new 2013-04-03 10:37:52.710636 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos 2013-04-03 10:37:52.710651 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 2013-04-03 10:37:52.710669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:52.721969 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos 2013-04-03 10:37:52.721985 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 2013-04-03 10:37:52.721999 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:52.928998 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:52.929006 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52600 from 10.252.24.88:6789/0 2013-04-03 10:37:52.929036 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 66 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:37:53.337529 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29023 check_down_pgs 2013-04-03 10:37:53.337563 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29023 v29023: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:37:53.337872 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:37:53.337896 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:37:53.337933 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:37:53.337964 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos 2013-04-03 10:37:53.337983 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 2013-04-03 10:37:53.337990 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394026 log 2013-04-03 10:37:53.338006 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:53.338024 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:37:53.382272 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21722 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:37:55.582862 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140352 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:37:55.711202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos 2013-04-03 10:37:55.711221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 2013-04-03 10:37:55.711238 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:56.928958 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:56.928968 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52b80 from 10.252.24.88:6789/0 2013-04-03 10:37:56.929019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 67 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:37:57.382445 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21723 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:37:58.215661 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:37:58.215671 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:37:58.215723 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 preprocess_query pg_stats(1 pgs tid 4 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:58.215729 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 prepare_update pg_stats(1 pgs tid 4 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:58.215732 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 prepare_pg_stats pg_stats(1 pgs tid 4 v 605) v1 from osd.2 2013-04-03 10:37:58.215739 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:37:58.265925 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29023 encode_pending v 29024 2013-04-03 10:37:58.293112 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29023 update_from_paxos applying incremental 29024 2013-04-03 10:37:58.293128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 v29024: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3239B/s wr, 0op/s 2013-04-03 10:37:58.293145 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 send_pg_creates to 0 pgs 2013-04-03 10:37:58.293148 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 update_logger 2013-04-03 10:37:58.293157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 create_pending v 29025 2013-04-03 10:37:58.293166 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29024 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:37:58.293184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 check_osd_map already seen 605 >= 605 2013-04-03 10:37:58.293186 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 update_logger 2013-04-03 10:37:58.293196 7fcdffcdb700 0 log [INF] : pgmap v29024: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3239B/s wr, 0op/s 2013-04-03 10:37:58.293269 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos 2013-04-03 10:37:58.293288 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 2013-04-03 10:37:58.293305 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:58.293357 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos 2013-04-03 10:37:58.293371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 2013-04-03 10:37:58.293379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:58.293389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:37:58.293418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:37:58.293429 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:37:58.293439 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 logging 2013-04-03 10:37:52.685389 mon.0 10.252.24.25:6789/0 89 : [INF] pgmap v29023: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:37:58.293458 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 logging 2013-04-03 10:37:58.293202 mon.0 10.252.24.25:6789/0 90 : [INF] pgmap v29024: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3239B/s wr, 0op/s 2013-04-03 10:37:58.317482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos 2013-04-03 10:37:58.317502 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 2013-04-03 10:37:58.317517 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:58.338135 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29024 check_down_pgs 2013-04-03 10:37:58.338169 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29024 v29024: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3239B/s wr, 0op/s 2013-04-03 10:37:58.338194 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:37:58.338215 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:37:58.338251 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:37:58.338281 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos 2013-04-03 10:37:58.338297 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 2013-04-03 10:37:58.338306 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394026 log 2013-04-03 10:37:58.338451 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:58.338474 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:37:59.292347 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394026 encode_pending v394027 2013-04-03 10:37:59.341961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos 2013-04-03 10:37:59.341990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394026 2013-04-03 10:37:59.342004 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos latest full 393798 2013-04-03 10:37:59.342020 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394027 update_from_paxos applying incremental log 394027 2013-04-03 10:37:52.685389 mon.0 10.252.24.25:6789/0 89 : [INF] pgmap v29023: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:37:59.342048 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394027 update_from_paxos applying incremental log 394027 2013-04-03 10:37:58.293202 mon.0 10.252.24.25:6789/0 90 : [INF] pgmap v29024: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3239B/s wr, 0op/s 2013-04-03 10:37:59.342086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 check_subs 2013-04-03 10:37:59.342117 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 create_pending v 394028 2013-04-03 10:37:59.342139 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394027 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:37:59.342202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos 2013-04-03 10:37:59.342218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 2013-04-03 10:37:59.342237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:59.353531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos 2013-04-03 10:37:59.353547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 2013-04-03 10:37:59.353561 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:37:59.582944 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140353 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:38:00.441216 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acda00 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33057/0) 2013-04-03 10:38:00.929030 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:00.929038 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2996340 from 10.252.24.88:6789/0 2013-04-03 10:38:00.929089 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 68 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:38:01.382565 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21724 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:38:02.342889 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos 2013-04-03 10:38:02.342911 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 2013-04-03 10:38:02.342928 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:03.338613 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29024 v29024: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3239B/s wr, 0op/s 2013-04-03 10:38:03.338684 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:38:03.338707 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:38:03.338749 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:38:03.338780 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos 2013-04-03 10:38:03.338800 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 2013-04-03 10:38:03.338808 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394027 log 2013-04-03 10:38:03.338826 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:03.338844 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:38:03.583079 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140354 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:38:04.929203 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:04.929212 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2995000 from 10.252.24.88:6789/0 2013-04-03 10:38:04.929264 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 69 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:38:05.342888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos 2013-04-03 10:38:05.342908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 2013-04-03 10:38:05.342924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:05.382612 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21725 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:38:06.579122 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:06.579132 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:38:06.579183 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 preprocess_query pg_stats(1 pgs tid 23 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:38:06.579206 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 prepare_update pg_stats(1 pgs tid 23 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:38:06.579212 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 prepare_pg_stats pg_stats(1 pgs tid 23 v 605) v1 from osd.1 2013-04-03 10:38:06.579218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:38:06.629407 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29024 encode_pending v 29025 2013-04-03 10:38:06.763618 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29024 update_from_paxos applying incremental 29025 2013-04-03 10:38:06.763638 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 v29025: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s 2013-04-03 10:38:06.763656 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 send_pg_creates to 0 pgs 2013-04-03 10:38:06.763659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 update_logger 2013-04-03 10:38:06.763669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 create_pending v 29026 2013-04-03 10:38:06.763678 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29025 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:38:06.763698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 check_osd_map already seen 605 >= 605 2013-04-03 10:38:06.763700 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 update_logger 2013-04-03 10:38:06.763710 7fcdffcdb700 0 log [INF] : pgmap v29025: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s 2013-04-03 10:38:06.763770 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos 2013-04-03 10:38:06.763789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 2013-04-03 10:38:06.763805 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:06.763855 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos 2013-04-03 10:38:06.763869 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 2013-04-03 10:38:06.763877 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:06.763887 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:38:06.763909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:06.763921 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:38:06.763931 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 logging 2013-04-03 10:38:06.763711 mon.0 10.252.24.25:6789/0 91 : [INF] pgmap v29025: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s 2013-04-03 10:38:06.776327 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos 2013-04-03 10:38:06.776344 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 2013-04-03 10:38:06.776358 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:07.583224 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140355 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:38:07.763333 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394027 encode_pending v394028 2013-04-03 10:38:07.949705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos 2013-04-03 10:38:07.949730 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394027 2013-04-03 10:38:07.949744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos latest full 393798 2013-04-03 10:38:07.949761 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394028 update_from_paxos applying incremental log 394028 2013-04-03 10:38:06.763711 mon.0 10.252.24.25:6789/0 91 : [INF] pgmap v29025: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s 2013-04-03 10:38:07.949807 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 check_subs 2013-04-03 10:38:07.949846 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 create_pending v 394029 2013-04-03 10:38:07.949869 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394028 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:38:07.949929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos 2013-04-03 10:38:07.949945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 2013-04-03 10:38:07.949966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:07.962547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos 2013-04-03 10:38:07.962563 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 2013-04-03 10:38:07.962577 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:08.338986 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29025 check_down_pgs 2013-04-03 10:38:08.339016 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29025 v29025: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s 2013-04-03 10:38:08.339043 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:38:08.339064 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:38:08.339101 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:38:08.339132 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos 2013-04-03 10:38:08.339154 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 2013-04-03 10:38:08.339163 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394028 log 2013-04-03 10:38:08.339180 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:08.339198 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:38:08.929311 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:08.929320 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2995580 from 10.252.24.88:6789/0 2013-04-03 10:38:08.929358 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 70 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:38:09.382768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21726 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:38:10.950552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos 2013-04-03 10:38:10.950696 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 2013-04-03 10:38:10.950723 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:11.583367 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140356 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:38:12.929469 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:12.929478 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29952c0 from 10.252.24.88:6789/0 2013-04-03 10:38:12.929531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 71 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:38:13.217811 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:13.217821 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 2013-04-03 10:38:13.217872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 preprocess_query pg_stats(1 pgs tid 5 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:38:13.217893 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 prepare_update pg_stats(1 pgs tid 5 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:38:13.217897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 prepare_pg_stats pg_stats(1 pgs tid 5 v 605) v1 from osd.2 2013-04-03 10:38:13.217904 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:38:13.268105 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29025 encode_pending v 29026 2013-04-03 10:38:13.324350 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29025 update_from_paxos applying incremental 29026 2013-04-03 10:38:13.324369 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 v29026: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:38:13.324387 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 send_pg_creates to 0 pgs 2013-04-03 10:38:13.324391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 update_logger 2013-04-03 10:38:13.324401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 create_pending v 29027 2013-04-03 10:38:13.324410 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29026 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:38:13.324425 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 check_osd_map already seen 605 >= 605 2013-04-03 10:38:13.324427 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 update_logger 2013-04-03 10:38:13.324437 7fcdffcdb700 0 log [INF] : pgmap v29026: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:38:13.324502 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos 2013-04-03 10:38:13.324521 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 2013-04-03 10:38:13.324539 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:13.324581 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos 2013-04-03 10:38:13.324596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 2013-04-03 10:38:13.324605 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:13.324615 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:38:13.324638 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:13.324649 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:38:13.324659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 logging 2013-04-03 10:38:13.324438 mon.0 10.252.24.25:6789/0 92 : [INF] pgmap v29026: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:38:13.339313 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29026 check_down_pgs 2013-04-03 10:38:13.339348 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29026 v29026: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:38:13.339369 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:38:13.339390 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:38:13.339426 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:38:13.339456 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos 2013-04-03 10:38:13.339473 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 2013-04-03 10:38:13.339482 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394028 log 2013-04-03 10:38:13.339497 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:13.339516 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:38:13.349351 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos 2013-04-03 10:38:13.349367 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 2013-04-03 10:38:13.349381 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:13.382813 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21727 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:38:14.324147 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394028 encode_pending v394029 2013-04-03 10:38:14.440444 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos 2013-04-03 10:38:14.440470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394028 2013-04-03 10:38:14.440484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos latest full 393798 2013-04-03 10:38:14.440500 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394029 update_from_paxos applying incremental log 394029 2013-04-03 10:38:13.324438 mon.0 10.252.24.25:6789/0 92 : [INF] pgmap v29026: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:38:14.440547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 check_subs 2013-04-03 10:38:14.440578 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 create_pending v 394030 2013-04-03 10:38:14.440600 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394029 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:38:14.440663 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos 2013-04-03 10:38:14.440678 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 2013-04-03 10:38:14.440696 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:14.451970 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos 2013-04-03 10:38:14.451990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 2013-04-03 10:38:14.452006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:15.583482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140357 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:38:16.929718 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:16.929728 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52b80 from 10.252.24.88:6789/0 2013-04-03 10:38:16.929778 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 72 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:38:17.383026 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21728 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:38:17.441424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos 2013-04-03 10:38:17.441443 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 2013-04-03 10:38:17.441460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:18.339664 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29026 v29026: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:38:18.339704 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:38:18.339727 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:38:18.339767 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:38:18.339798 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos 2013-04-03 10:38:18.339816 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 2013-04-03 10:38:18.339826 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394029 log 2013-04-03 10:38:18.339842 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:18.339861 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:38:19.583601 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140358 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:38:20.441387 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos 2013-04-03 10:38:20.441407 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 2013-04-03 10:38:20.441424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:20.929652 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:20.929660 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52600 from 10.252.24.88:6789/0 2013-04-03 10:38:20.929693 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 73 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:38:21.383212 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21729 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:38:21.580827 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:21.580835 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:38:21.580869 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 preprocess_query pg_stats(1 pgs tid 24 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:38:21.580892 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 prepare_update pg_stats(1 pgs tid 24 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:38:21.580898 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 prepare_pg_stats pg_stats(1 pgs tid 24 v 605) v1 from osd.1 2013-04-03 10:38:21.580907 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:38:21.631090 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29026 encode_pending v 29027 2013-04-03 10:38:21.656852 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29026 update_from_paxos applying incremental 29027 2013-04-03 10:38:21.656867 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 v29027: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:38:21.656885 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 send_pg_creates to 0 pgs 2013-04-03 10:38:21.656887 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 update_logger 2013-04-03 10:38:21.656897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 create_pending v 29028 2013-04-03 10:38:21.656905 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29027 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:38:21.656920 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 check_osd_map already seen 605 >= 605 2013-04-03 10:38:21.656922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 update_logger 2013-04-03 10:38:21.656932 7fcdffcdb700 0 log [INF] : pgmap v29027: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:38:21.656993 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos 2013-04-03 10:38:21.657014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 2013-04-03 10:38:21.657031 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:21.657070 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos 2013-04-03 10:38:21.657085 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 2013-04-03 10:38:21.657093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:21.657103 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:38:21.657125 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:21.657137 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:38:21.657146 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 logging 2013-04-03 10:38:21.656933 mon.0 10.252.24.25:6789/0 93 : [INF] pgmap v29027: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:38:21.667874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos 2013-04-03 10:38:21.667890 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 2013-04-03 10:38:21.667905 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:21.746778 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:21.746786 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 2013-04-03 10:38:21.746813 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 preprocess_query pg_stats(1 pgs tid 30443 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:38:21.746833 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 prepare_update pg_stats(1 pgs tid 30443 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:38:21.746840 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 prepare_pg_stats pg_stats(1 pgs tid 30443 v 605) v1 from osd.3 2013-04-03 10:38:21.746844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:38:22.656084 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394029 encode_pending v394030 2013-04-03 10:38:22.657298 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29027 encode_pending v 29028 2013-04-03 10:38:22.767352 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos 2013-04-03 10:38:22.767379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos version 394030 summary v 394029 2013-04-03 10:38:22.767393 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos latest full 393798 2013-04-03 10:38:22.767412 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394030 update_from_paxos applying incremental log 394030 2013-04-03 10:38:21.656933 mon.0 10.252.24.25:6789/0 93 : [INF] pgmap v29027: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:38:22.767457 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 check_subs 2013-04-03 10:38:22.767488 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 create_pending v 394031 2013-04-03 10:38:22.767510 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394030 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:38:22.793590 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29027 update_from_paxos applying incremental 29028 2013-04-03 10:38:22.793609 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 v29028: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:38:22.793627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 send_pg_creates to 0 pgs 2013-04-03 10:38:22.793631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 update_logger 2013-04-03 10:38:22.793641 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 create_pending v 29029 2013-04-03 10:38:22.793650 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29028 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:38:22.793665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 check_osd_map already seen 605 >= 605 2013-04-03 10:38:22.793667 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 update_logger 2013-04-03 10:38:22.793676 7fcdffcdb700 0 log [INF] : pgmap v29028: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:38:22.793735 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos 2013-04-03 10:38:22.793752 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos version 394030 summary v 394030 2013-04-03 10:38:22.793773 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:22.793815 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos 2013-04-03 10:38:22.793829 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos version 394030 summary v 394030 2013-04-03 10:38:22.793838 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:22.793848 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:38:22.793867 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:22.793878 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:38:22.793888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 logging 2013-04-03 10:38:22.793677 mon.0 10.252.24.25:6789/0 94 : [INF] pgmap v29028: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:38:22.804113 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos 2013-04-03 10:38:22.804129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos version 394030 summary v 394030 2013-04-03 10:38:22.804146 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:23.340003 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29028 check_down_pgs 2013-04-03 10:38:23.340035 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29028 v29028: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:38:23.340062 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:38:23.340084 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:38:23.340249 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:38:23.340283 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos 2013-04-03 10:38:23.340301 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos version 394030 summary v 394030 2013-04-03 10:38:23.340309 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394030 log 2013-04-03 10:38:23.340325 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:23.340341 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:38:23.583719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140359 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:38:23.792282 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394030 encode_pending v394031 2013-04-03 10:38:23.829001 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos 2013-04-03 10:38:23.829028 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394030 2013-04-03 10:38:23.829042 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos latest full 393798 2013-04-03 10:38:23.829058 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394031 update_from_paxos applying incremental log 394031 2013-04-03 10:38:22.793677 mon.0 10.252.24.25:6789/0 94 : [INF] pgmap v29028: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:38:23.829102 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 check_subs 2013-04-03 10:38:23.829133 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 create_pending v 394032 2013-04-03 10:38:23.829155 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394031 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:38:23.829216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos 2013-04-03 10:38:23.829232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394031 2013-04-03 10:38:23.829248 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:23.854105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos 2013-04-03 10:38:23.854121 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394031 2013-04-03 10:38:23.854136 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:24.929849 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:24.929859 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a512c0 from 10.252.24.88:6789/0 2013-04-03 10:38:24.929909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 74 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:38:25.383283 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21730 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:38:26.829922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos 2013-04-03 10:38:26.829942 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394031 2013-04-03 10:38:26.829959 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:27.583848 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140360 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:38:28.219697 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:28.219707 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 2013-04-03 10:38:28.219758 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 preprocess_query pg_stats(1 pgs tid 6 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:38:28.219784 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 prepare_update pg_stats(1 pgs tid 6 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:38:28.219793 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 prepare_pg_stats pg_stats(1 pgs tid 6 v 605) v1 from osd.2 2013-04-03 10:38:28.219799 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:38:28.269985 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29028 encode_pending v 29029 2013-04-03 10:38:28.299232 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29028 update_from_paxos applying incremental 29029 2013-04-03 10:38:28.299248 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29029 v29029: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1388B/s wr, 0op/s 2013-04-03 10:38:28.299266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29029 send_pg_creates to 0 pgs 2013-04-03 10:38:28.299271 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29029 update_logger 2013-04-03 10:38:28.299281 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29029 create_pending v 29030 2013-04-03 10:38:28.299289 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29029 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:38:28.299307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29029 check_osd_map already seen 605 >= 605 2013-04-03 10:38:28.299310 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29029 update_logger 2013-04-03 10:38:28.299320 7fcdffcdb700 0 log [INF] : pgmap v29029: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1388B/s wr, 0op/s 2013-04-03 10:38:28.299380 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos 2013-04-03 10:38:28.299399 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394031 2013-04-03 10:38:28.299417 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:28.299459 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos 2013-04-03 10:38:28.299473 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394031 2013-04-03 10:38:28.299482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:28.299493 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:38:28.299515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:28.299527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:38:28.299536 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 logging 2013-04-03 10:38:28.299321 mon.0 10.252.24.25:6789/0 95 : [INF] pgmap v29029: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1388B/s wr, 0op/s 2013-04-03 10:38:28.312082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos 2013-04-03 10:38:28.312098 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394031 2013-04-03 10:38:28.312112 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:28.340450 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29029 check_down_pgs 2013-04-03 10:38:28.340481 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29029 v29029: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1388B/s wr, 0op/s 2013-04-03 10:38:28.340508 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:38:28.340529 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:38:28.340566 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:38:28.340595 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos 2013-04-03 10:38:28.340612 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394031 2013-04-03 10:38:28.340620 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394031 log 2013-04-03 10:38:28.340635 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos 2013-04-03 10:38:28.340652 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth 2013-04-03 10:38:28.429070 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:38:28.429077 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:38:28.429086 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 2013-04-03 10:38:28.929938 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:28.929948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2145000 from 10.252.24.88:6789/0 2013-04-03 10:38:28.929999 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 75 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:38:29.298989 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394031 encode_pending v394032 2013-04-03 10:38:29.327188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos 2013-04-03 10:38:29.327215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394031 2013-04-03 10:38:29.327230 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos latest full 393798 2013-04-03 10:38:29.327247 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos applying incremental log 394032 2013-04-03 10:38:28.299321 mon.0 10.252.24.25:6789/0 95 : [INF] pgmap v29029: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1388B/s wr, 0op/s 2013-04-03 10:38:29.327294 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 check_subs 2013-04-03 10:38:29.327326 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 create_pending v 394033 2013-04-03 10:38:29.327349 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:38:29.327413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos 2013-04-03 10:38:29.327429 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 2013-04-03 10:38:29.327451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:29.352047 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos 2013-04-03 10:38:29.352063 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 2013-04-03 10:38:29.352078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:29.383337 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21731 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:38:31.583995 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140361 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:38:32.328106 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos 2013-04-03 10:38:32.328126 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 2013-04-03 10:38:32.328143 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:32.930050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:32.930061 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29c5600 from 10.252.24.88:6789/0 2013-04-03 10:38:32.930106 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 76 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:38:33.340794 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29029 v29029: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1388B/s wr, 0op/s 2013-04-03 10:38:33.340837 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:38:33.340863 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:38:33.340908 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:38:33.340939 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos 2013-04-03 10:38:33.340957 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 2013-04-03 10:38:33.340967 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 log 2013-04-03 10:38:33.340984 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:33.341002 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:38:33.383454 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21732 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:38:35.328164 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos 2013-04-03 10:38:35.328184 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 2013-04-03 10:38:35.328201 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:35.584663 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140362 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:38:36.582537 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:36.582548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:38:36.582599 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29029 preprocess_query pg_stats(1 pgs tid 25 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:38:36.582623 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29029 prepare_update pg_stats(1 pgs tid 25 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:38:36.582633 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29029 prepare_pg_stats pg_stats(1 pgs tid 25 v 605) v1 from osd.1 2013-04-03 10:38:36.582640 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29029 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:38:36.632832 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29029 encode_pending v 29030 2013-04-03 10:38:36.658107 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29029 update_from_paxos applying incremental 29030 2013-04-03 10:38:36.658123 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 v29030: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 879B/s wr, 0op/s 2013-04-03 10:38:36.658144 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 send_pg_creates to 0 pgs 2013-04-03 10:38:36.658147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 update_logger 2013-04-03 10:38:36.658157 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 create_pending v 29031 2013-04-03 10:38:36.658166 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:38:36.658182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 check_osd_map already seen 605 >= 605 2013-04-03 10:38:36.658185 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 update_logger 2013-04-03 10:38:36.658196 7fcdffcdb700 0 log [INF] : pgmap v29030: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 879B/s wr, 0op/s 2013-04-03 10:38:36.658263 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos 2013-04-03 10:38:36.658282 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 2013-04-03 10:38:36.658301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:36.658348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos 2013-04-03 10:38:36.658363 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 2013-04-03 10:38:36.658372 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:36.658384 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:38:36.658409 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:36.658423 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:38:36.658433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 logging 2013-04-03 10:38:36.658197 mon.0 10.252.24.25:6789/0 96 : [INF] pgmap v29030: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 879B/s wr, 0op/s 2013-04-03 10:38:36.687666 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos 2013-04-03 10:38:36.687682 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 2013-04-03 10:38:36.687697 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:36.930129 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:36.930139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29c5080 from 10.252.24.88:6789/0 2013-04-03 10:38:36.930171 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 77 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:38:37.383558 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21733 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:38:37.657854 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 encode_pending v394033 2013-04-03 10:38:37.719340 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos 2013-04-03 10:38:37.719367 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394032 2013-04-03 10:38:37.719382 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos latest full 393798 2013-04-03 10:38:37.719399 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos applying incremental log 394033 2013-04-03 10:38:36.658197 mon.0 10.252.24.25:6789/0 96 : [INF] pgmap v29030: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 879B/s wr, 0op/s 2013-04-03 10:38:37.719457 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 check_subs 2013-04-03 10:38:37.719490 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 create_pending v 394034 2013-04-03 10:38:37.719513 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:38:37.719578 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos 2013-04-03 10:38:37.719593 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394033 2013-04-03 10:38:37.719612 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:37.732385 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos 2013-04-03 10:38:37.732402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394033 2013-04-03 10:38:37.732417 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:38.341144 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 check_down_pgs 2013-04-03 10:38:38.341176 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 v29030: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 879B/s wr, 0op/s 2013-04-03 10:38:38.341203 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:38:38.341228 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:38:38.341265 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:38:38.341297 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos 2013-04-03 10:38:38.341328 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394033 2013-04-03 10:38:38.341338 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 log 2013-04-03 10:38:38.341356 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:38.341375 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:38:39.584218 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140363 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:38:40.720307 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos 2013-04-03 10:38:40.720327 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394033 2013-04-03 10:38:40.720345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:40.930293 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:40.930302 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29c4840 from 10.252.24.88:6789/0 2013-04-03 10:38:40.930334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 78 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:38:41.383715 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21734 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:38:41.583396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:41.583407 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 2013-04-03 10:38:41.583457 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 preprocess_query pg_stats(1 pgs tid 26 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:38:41.583481 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 prepare_update pg_stats(1 pgs tid 26 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:38:41.583488 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 prepare_pg_stats pg_stats(1 pgs tid 26 v 605) v1 from osd.1 2013-04-03 10:38:41.583494 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:38:41.633681 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 encode_pending v 29031 2013-04-03 10:38:41.662386 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 update_from_paxos applying incremental 29031 2013-04-03 10:38:41.662403 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 v29031: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 383B/s wr, 0op/s 2013-04-03 10:38:41.662424 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 send_pg_creates to 0 pgs 2013-04-03 10:38:41.662428 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 update_logger 2013-04-03 10:38:41.662439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 create_pending v 29032 2013-04-03 10:38:41.662448 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:38:41.662470 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 check_osd_map already seen 605 >= 605 2013-04-03 10:38:41.662474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 update_logger 2013-04-03 10:38:41.662484 7fcdffcdb700 0 log [INF] : pgmap v29031: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 383B/s wr, 0op/s 2013-04-03 10:38:41.662727 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos 2013-04-03 10:38:41.662749 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394033 2013-04-03 10:38:41.662768 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:41.662812 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos 2013-04-03 10:38:41.662827 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394033 2013-04-03 10:38:41.662836 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:41.662848 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:38:41.662873 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:41.662886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:38:41.662898 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 logging 2013-04-03 10:38:41.662485 mon.0 10.252.24.25:6789/0 97 : [INF] pgmap v29031: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 383B/s wr, 0op/s 2013-04-03 10:38:41.674437 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos 2013-04-03 10:38:41.674453 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394033 2013-04-03 10:38:41.674468 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:41.748933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:41.748941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:38:41.748970 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 preprocess_query pg_stats(1 pgs tid 30444 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:38:41.748991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 prepare_update pg_stats(1 pgs tid 30444 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:38:41.748997 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 prepare_pg_stats pg_stats(1 pgs tid 30444 v 605) v1 from osd.3 2013-04-03 10:38:41.749003 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 got osd.3 osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:38:42.661351 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 encode_pending v394034 2013-04-03 10:38:42.662542 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 encode_pending v 29032 2013-04-03 10:38:42.769277 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos 2013-04-03 10:38:42.769329 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos version 394034 summary v 394033 2013-04-03 10:38:42.769344 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos latest full 393798 2013-04-03 10:38:42.769363 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos applying incremental log 394034 2013-04-03 10:38:41.662485 mon.0 10.252.24.25:6789/0 97 : [INF] pgmap v29031: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 383B/s wr, 0op/s 2013-04-03 10:38:42.769450 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 check_subs 2013-04-03 10:38:42.769485 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 create_pending v 394035 2013-04-03 10:38:42.769508 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:38:42.806132 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 update_from_paxos applying incremental 29032 2013-04-03 10:38:42.806152 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 v29032: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 339B/s wr, 0op/s 2013-04-03 10:38:42.806172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 send_pg_creates to 0 pgs 2013-04-03 10:38:42.806176 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 update_logger 2013-04-03 10:38:42.806187 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 create_pending v 29033 2013-04-03 10:38:42.806198 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:38:42.806215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 check_osd_map already seen 605 >= 605 2013-04-03 10:38:42.806221 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 update_logger 2013-04-03 10:38:42.806231 7fcdffcdb700 0 log [INF] : pgmap v29032: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 339B/s wr, 0op/s 2013-04-03 10:38:42.806292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos 2013-04-03 10:38:42.806312 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos version 394034 summary v 394034 2013-04-03 10:38:42.806331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:42.806373 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos 2013-04-03 10:38:42.806389 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos version 394034 summary v 394034 2013-04-03 10:38:42.806399 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:42.806411 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:38:42.806435 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:42.806449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:38:42.806460 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 logging 2013-04-03 10:38:42.806232 mon.0 10.252.24.25:6789/0 98 : [INF] pgmap v29032: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 339B/s wr, 0op/s 2013-04-03 10:38:42.819391 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos 2013-04-03 10:38:42.819410 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos version 394034 summary v 394034 2013-04-03 10:38:42.819427 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:43.222069 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:43.222081 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 2013-04-03 10:38:43.222133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 preprocess_query pg_stats(1 pgs tid 7 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:38:43.222158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 prepare_update pg_stats(1 pgs tid 7 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:38:43.222165 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 prepare_pg_stats pg_stats(1 pgs tid 7 v 605) v1 from osd.2 2013-04-03 10:38:43.222172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:38:43.341505 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 check_down_pgs 2013-04-03 10:38:43.341538 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 v29032: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 339B/s wr, 0op/s 2013-04-03 10:38:43.341567 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:38:43.341589 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:38:43.341625 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:38:43.341659 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos 2013-04-03 10:38:43.341679 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos version 394034 summary v 394034 2013-04-03 10:38:43.341688 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 log 2013-04-03 10:38:43.341705 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:43.341724 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:38:43.584306 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140364 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:38:43.805870 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 encode_pending v394035 2013-04-03 10:38:43.807120 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 encode_pending v 29033 2013-04-03 10:38:43.839462 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos 2013-04-03 10:38:43.839488 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos version 394035 summary v 394034 2013-04-03 10:38:43.839504 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos latest full 393798 2013-04-03 10:38:43.839523 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos applying incremental log 394035 2013-04-03 10:38:42.806232 mon.0 10.252.24.25:6789/0 98 : [INF] pgmap v29032: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 339B/s wr, 0op/s 2013-04-03 10:38:43.839574 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 check_subs 2013-04-03 10:38:43.839606 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 create_pending v 394036 2013-04-03 10:38:43.839629 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:38:43.865629 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 update_from_paxos applying incremental 29033 2013-04-03 10:38:43.865648 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 v29033: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 8009B/s wr, 0op/s 2013-04-03 10:38:43.865668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 send_pg_creates to 0 pgs 2013-04-03 10:38:43.865671 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 update_logger 2013-04-03 10:38:43.865681 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 create_pending v 29034 2013-04-03 10:38:43.865691 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:38:43.865711 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 check_osd_map already seen 605 >= 605 2013-04-03 10:38:43.865714 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 update_logger 2013-04-03 10:38:43.865725 7fcdffcdb700 0 log [INF] : pgmap v29033: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 8009B/s wr, 0op/s 2013-04-03 10:38:43.865782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos 2013-04-03 10:38:43.865799 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos version 394035 summary v 394035 2013-04-03 10:38:43.865819 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:43.865861 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos 2013-04-03 10:38:43.865876 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos version 394035 summary v 394035 2013-04-03 10:38:43.865886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:43.865899 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:38:43.865918 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:43.865931 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:38:43.865941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 logging 2013-04-03 10:38:43.865726 mon.0 10.252.24.25:6789/0 99 : [INF] pgmap v29033: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 8009B/s wr, 0op/s 2013-04-03 10:38:43.890421 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos 2013-04-03 10:38:43.890439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos version 394035 summary v 394035 2013-04-03 10:38:43.890455 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:44.864415 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 encode_pending v394036 2013-04-03 10:38:44.897822 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos 2013-04-03 10:38:44.897850 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394035 2013-04-03 10:38:44.897865 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos latest full 393798 2013-04-03 10:38:44.897885 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos applying incremental log 394036 2013-04-03 10:38:43.865726 mon.0 10.252.24.25:6789/0 99 : [INF] pgmap v29033: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 8009B/s wr, 0op/s 2013-04-03 10:38:44.897930 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 check_subs 2013-04-03 10:38:44.897962 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 create_pending v 394037 2013-04-03 10:38:44.897985 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:38:44.898049 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos 2013-04-03 10:38:44.898065 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 2013-04-03 10:38:44.898084 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:44.911210 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos 2013-04-03 10:38:44.911227 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 2013-04-03 10:38:44.911243 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:44.930355 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:44.930364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273a580 from 10.252.24.88:6789/0 2013-04-03 10:38:44.930395 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 79 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:38:45.383857 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21735 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:38:47.416510 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 2013-04-03 10:38:47.416522 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 2013-04-03 10:38:47.584438 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140365 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:38:47.898683 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos 2013-04-03 10:38:47.898704 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 2013-04-03 10:38:47.898721 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:48.341866 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 check_down_pgs 2013-04-03 10:38:48.341899 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 v29033: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 8009B/s wr, 0op/s 2013-04-03 10:38:48.341927 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:38:48.341952 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:38:48.341989 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:38:48.342021 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos 2013-04-03 10:38:48.342040 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 2013-04-03 10:38:48.342049 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 log 2013-04-03 10:38:48.342066 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:48.342085 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:38:48.930590 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:48.930601 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273b340 from 10.252.24.88:6789/0 2013-04-03 10:38:48.930653 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 80 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:38:49.383951 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21736 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:38:50.521262 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33065/0) 2013-04-03 10:38:50.898748 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos 2013-04-03 10:38:50.898769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 2013-04-03 10:38:50.898786 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:51.584538 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140366 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:38:51.584571 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:51.584580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:38:51.584610 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 preprocess_query pg_stats(1 pgs tid 27 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:38:51.584633 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 prepare_update pg_stats(1 pgs tid 27 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:38:51.584640 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 prepare_pg_stats pg_stats(1 pgs tid 27 v 605) v1 from osd.1 2013-04-03 10:38:51.584646 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:38:51.634832 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 encode_pending v 29034 2013-04-03 10:38:51.674075 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 update_from_paxos applying incremental 29034 2013-04-03 10:38:51.674092 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 v29034: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2282B/s wr, 0op/s 2013-04-03 10:38:51.674113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 send_pg_creates to 0 pgs 2013-04-03 10:38:51.674120 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 update_logger 2013-04-03 10:38:51.674131 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 create_pending v 29035 2013-04-03 10:38:51.674141 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:38:51.674158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 check_osd_map already seen 605 >= 605 2013-04-03 10:38:51.674162 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 update_logger 2013-04-03 10:38:51.674173 7fcdffcdb700 0 log [INF] : pgmap v29034: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2282B/s wr, 0op/s 2013-04-03 10:38:51.674227 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos 2013-04-03 10:38:51.674247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 2013-04-03 10:38:51.674266 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:51.674308 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos 2013-04-03 10:38:51.674324 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 2013-04-03 10:38:51.674333 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:51.674345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:38:51.674374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:51.674388 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:38:51.674398 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 logging 2013-04-03 10:38:51.674174 mon.0 10.252.24.25:6789/0 100 : [INF] pgmap v29034: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2282B/s wr, 0op/s 2013-04-03 10:38:51.699963 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos 2013-04-03 10:38:51.699984 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 2013-04-03 10:38:51.700001 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:52.673879 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 encode_pending v394037 2013-04-03 10:38:52.701391 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos 2013-04-03 10:38:52.701428 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394036 2013-04-03 10:38:52.701443 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos latest full 393798 2013-04-03 10:38:52.701462 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos applying incremental log 394037 2013-04-03 10:38:51.674174 mon.0 10.252.24.25:6789/0 100 : [INF] pgmap v29034: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2282B/s wr, 0op/s 2013-04-03 10:38:52.701515 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 check_subs 2013-04-03 10:38:52.701548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 create_pending v 394038 2013-04-03 10:38:52.701571 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:38:52.701636 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos 2013-04-03 10:38:52.701655 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 2013-04-03 10:38:52.701673 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:52.713357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos 2013-04-03 10:38:52.713375 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 2013-04-03 10:38:52.713391 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:52.930656 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:52.930665 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc5b80 from 10.252.24.88:6789/0 2013-04-03 10:38:52.930696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 81 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:38:53.332351 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_start_round curr 2 2013-04-03 10:38:53.332379 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_start_round new 3 2013-04-03 10:38:53.332385 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck 2013-04-03 10:38:53.332388 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck start timecheck epoch 352 round 3 2013-04-03 10:38:53.332400 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck send time_check( ping e 352 r 3 ) v1 to mon.2 10.252.24.88:6789/0 2013-04-03 10:38:53.332435 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_start_round setting up next event 2013-04-03 10:38:53.333188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck time_check( pong e 352 r 3 ts 2013-04-03 10:38:53.332248 ) v1 2013-04-03 10:38:53.333195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck_leader time_check( pong e 352 r 3 ts 2013-04-03 10:38:53.332248 ) v1 2013-04-03 10:38:53.333203 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck_leader from mon.2 10.252.24.88:6789/0 ts 2013-04-03 10:38:53.332248 delta -0.000951052 skew_bound -0.000149052 latency 0.000802 2013-04-03 10:38:53.333219 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck_leader got pongs from everybody (2 total) 2013-04-03 10:38:53.333224 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_finish_round curr 3 2013-04-03 10:38:53.333226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_report 2013-04-03 10:38:53.333231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_report send report to mon.2 10.252.24.88:6789/0 2013-04-03 10:38:53.342225 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 check_down_pgs 2013-04-03 10:38:53.342257 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 v29034: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2282B/s wr, 0op/s 2013-04-03 10:38:53.342287 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:38:53.342313 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:38:53.342692 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:38:53.342728 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos 2013-04-03 10:38:53.342747 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 2013-04-03 10:38:53.342757 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 log 2013-04-03 10:38:53.342774 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos 2013-04-03 10:38:53.342793 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth 2013-04-03 10:38:53.342826 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 AuthMonitor::tick() updated rotating, now calling propose_pending 2013-04-03 10:38:53.342891 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 encode_pending v 2061 2013-04-03 10:38:53.380884 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:38:53.380918 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos version 2061 keys ver 2060 latest 2059 2013-04-03 10:38:53.380928 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos key server version 2060 2013-04-03 10:38:53.380960 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos() last_allocated_id=11896 max_global_id=11896 2013-04-03 10:38:53.380979 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 create_pending v 2062 2013-04-03 10:38:53.381001 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 AuthMonitor::on_active() 2013-04-03 10:38:53.381047 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos 2013-04-03 10:38:53.381064 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 2013-04-03 10:38:53.381081 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:38:53.383994 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21737 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:38:53.406738 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos 2013-04-03 10:38:53.406756 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 2013-04-03 10:38:53.406773 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:38:55.481050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 2013-04-03 10:38:55.481061 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 2013-04-03 10:38:55.584672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140367 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:38:56.381761 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos 2013-04-03 10:38:56.381782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 2013-04-03 10:38:56.381800 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:38:56.930864 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:56.930875 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc58c0 from 10.252.24.88:6789/0 2013-04-03 10:38:56.930926 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 82 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:38:57.384129 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21738 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:38:58.223559 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:38:58.223570 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:38:58.223623 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 preprocess_query pg_stats(1 pgs tid 8 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:38:58.223658 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 prepare_update pg_stats(1 pgs tid 8 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:38:58.223666 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 prepare_pg_stats pg_stats(1 pgs tid 8 v 605) v1 from osd.2 2013-04-03 10:38:58.223673 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:38:58.273815 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 encode_pending v 29035 2013-04-03 10:38:58.320033 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 update_from_paxos applying incremental 29035 2013-04-03 10:38:58.320050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 v29035: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1132B/s wr, 0op/s 2013-04-03 10:38:58.320071 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 send_pg_creates to 0 pgs 2013-04-03 10:38:58.320075 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 update_logger 2013-04-03 10:38:58.320086 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 create_pending v 29036 2013-04-03 10:38:58.320097 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:38:58.320113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 check_osd_map already seen 605 >= 605 2013-04-03 10:38:58.320116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 update_logger 2013-04-03 10:38:58.320126 7fcdffcdb700 0 log [INF] : pgmap v29035: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1132B/s wr, 0op/s 2013-04-03 10:38:58.320186 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos 2013-04-03 10:38:58.320206 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 2013-04-03 10:38:58.320225 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:38:58.320268 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos 2013-04-03 10:38:58.320283 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 2013-04-03 10:38:58.320292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:58.320304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:38:58.320328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:38:58.320341 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:38:58.320354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 logging 2013-04-03 10:38:58.320128 mon.0 10.252.24.25:6789/0 101 : [INF] pgmap v29035: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1132B/s wr, 0op/s 2013-04-03 10:38:58.332040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos 2013-04-03 10:38:58.332056 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 2013-04-03 10:38:58.332073 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:38:58.344630 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 check_down_pgs 2013-04-03 10:38:58.344666 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 v29035: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1132B/s wr, 0op/s 2013-04-03 10:38:58.344692 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:38:58.344713 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:38:58.344751 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:38:58.344781 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos 2013-04-03 10:38:58.344799 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 2013-04-03 10:38:58.344808 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 log 2013-04-03 10:38:58.344824 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:38:58.344842 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:38:58.429147 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) 2013-04-03 10:38:58.429172 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader 2013-04-03 10:38:59.319842 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394037 encode_pending v394038 2013-04-03 10:38:59.434523 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos 2013-04-03 10:38:59.434550 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394037 2013-04-03 10:38:59.434564 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos latest full 393798 2013-04-03 10:38:59.434582 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394038 update_from_paxos applying incremental log 394038 2013-04-03 10:38:58.320128 mon.0 10.252.24.25:6789/0 101 : [INF] pgmap v29035: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1132B/s wr, 0op/s 2013-04-03 10:38:59.434628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 check_subs 2013-04-03 10:38:59.434659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 create_pending v 394039 2013-04-03 10:38:59.434681 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394038 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:38:59.434744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos 2013-04-03 10:38:59.434760 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 2013-04-03 10:38:59.434777 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:38:59.460371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos 2013-04-03 10:38:59.460802 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 2013-04-03 10:38:59.460826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:38:59.584788 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140368 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:00.537219 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd500 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33066/0) 2013-04-03 10:39:00.930973 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:00.930984 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc5340 from 10.252.24.88:6789/0 2013-04-03 10:39:00.931036 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 83 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:39:01.384270 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21739 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:39:02.435445 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos 2013-04-03 10:39:02.435465 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 2013-04-03 10:39:02.435482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:03.344982 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29035 v29035: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1132B/s wr, 0op/s 2013-04-03 10:39:03.345031 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:39:03.345056 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:39:03.345100 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:39:03.345134 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos 2013-04-03 10:39:03.345153 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 2013-04-03 10:39:03.345161 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394038 log 2013-04-03 10:39:03.345177 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:03.345196 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:39:03.584974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140369 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:04.931111 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:04.931121 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc5080 from 10.252.24.88:6789/0 2013-04-03 10:39:04.931173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 84 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:39:05.384358 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21740 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:39:05.435448 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos 2013-04-03 10:39:05.435468 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 2013-04-03 10:39:05.435486 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:06.586274 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:06.586284 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 2013-04-03 10:39:06.586336 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29035 preprocess_query pg_stats(1 pgs tid 28 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:06.586360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29035 prepare_update pg_stats(1 pgs tid 28 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:06.586365 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29035 prepare_pg_stats pg_stats(1 pgs tid 28 v 605) v1 from osd.1 2013-04-03 10:39:06.586371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29035 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:39:06.636556 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29035 encode_pending v 29036 2013-04-03 10:39:06.672806 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29035 update_from_paxos applying incremental 29036 2013-04-03 10:39:06.672823 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 v29036: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:39:06.672842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 send_pg_creates to 0 pgs 2013-04-03 10:39:06.672847 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 update_logger 2013-04-03 10:39:06.672857 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 create_pending v 29037 2013-04-03 10:39:06.672866 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29036 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:06.672884 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 check_osd_map already seen 605 >= 605 2013-04-03 10:39:06.672887 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 update_logger 2013-04-03 10:39:06.672897 7fcdffcdb700 0 log [INF] : pgmap v29036: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:39:06.672956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos 2013-04-03 10:39:06.672975 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 2013-04-03 10:39:06.672993 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:06.673036 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos 2013-04-03 10:39:06.673050 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 2013-04-03 10:39:06.673058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:06.673068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:39:06.673092 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:06.673105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:39:06.673114 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 logging 2013-04-03 10:39:06.672898 mon.0 10.252.24.25:6789/0 102 : [INF] pgmap v29036: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:39:06.687896 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos 2013-04-03 10:39:06.687913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 2013-04-03 10:39:06.687929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:06.751641 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:06.751649 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:39:06.751678 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 preprocess_query pg_stats(1 pgs tid 30445 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:39:06.751698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 prepare_update pg_stats(1 pgs tid 30445 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:39:06.751704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 prepare_pg_stats pg_stats(1 pgs tid 30445 v 605) v1 from osd.3 2013-04-03 10:39:06.751708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 got osd.3 osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:39:07.585084 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140370 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:07.666188 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29036 encode_pending v 29037 2013-04-03 10:39:07.668132 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394038 encode_pending v394039 2013-04-03 10:39:07.697809 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29036 update_from_paxos applying incremental 29037 2013-04-03 10:39:07.697837 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 v29037: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:39:07.697858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 send_pg_creates to 0 pgs 2013-04-03 10:39:07.697861 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 update_logger 2013-04-03 10:39:07.697872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 create_pending v 29038 2013-04-03 10:39:07.697881 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29037 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:39:07.697897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 check_osd_map already seen 605 >= 605 2013-04-03 10:39:07.697900 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 update_logger 2013-04-03 10:39:07.697911 7fcdffcdb700 0 log [INF] : pgmap v29037: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:39:07.737829 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos 2013-04-03 10:39:07.737852 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394038 2013-04-03 10:39:07.737866 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos latest full 393798 2013-04-03 10:39:07.737884 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394039 update_from_paxos applying incremental log 394039 2013-04-03 10:39:06.672898 mon.0 10.252.24.25:6789/0 102 : [INF] pgmap v29036: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:39:07.737930 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 check_subs 2013-04-03 10:39:07.737961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 create_pending v 394040 2013-04-03 10:39:07.738257 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394039 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:39:07.738291 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos 2013-04-03 10:39:07.738305 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 2013-04-03 10:39:07.738315 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:07.738327 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:39:07.738345 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 nothing new 2013-04-03 10:39:07.738391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos 2013-04-03 10:39:07.738406 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 2013-04-03 10:39:07.738424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:07.751211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos 2013-04-03 10:39:07.751228 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 2013-04-03 10:39:07.751244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:08.345350 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29037 check_down_pgs 2013-04-03 10:39:08.345379 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29037 v29037: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:39:08.345410 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:39:08.345434 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:39:08.345470 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:39:08.345501 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos 2013-04-03 10:39:08.345999 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 2013-04-03 10:39:08.346017 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394039 log 2013-04-03 10:39:08.346035 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:08.346054 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:39:08.931249 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:08.931261 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4840 from 10.252.24.88:6789/0 2013-04-03 10:39:08.931314 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 85 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:39:09.384461 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21741 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:39:10.738740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos 2013-04-03 10:39:10.738759 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 2013-04-03 10:39:10.738776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:11.585206 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140371 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:12.931415 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:12.931425 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4580 from 10.252.24.88:6789/0 2013-04-03 10:39:12.931479 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 86 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:39:13.225848 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:13.225858 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 2013-04-03 10:39:13.225910 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 preprocess_query pg_stats(1 pgs tid 9 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:39:13.225934 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 prepare_update pg_stats(1 pgs tid 9 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:39:13.225940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 prepare_pg_stats pg_stats(1 pgs tid 9 v 605) v1 from osd.2 2013-04-03 10:39:13.225946 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:39:13.276136 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29037 encode_pending v 29038 2013-04-03 10:39:13.308089 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29037 update_from_paxos applying incremental 29038 2013-04-03 10:39:13.308105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 v29038: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2621B/s wr, 0op/s 2013-04-03 10:39:13.308125 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 send_pg_creates to 0 pgs 2013-04-03 10:39:13.308127 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 update_logger 2013-04-03 10:39:13.308139 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 create_pending v 29039 2013-04-03 10:39:13.308152 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29038 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:39:13.308166 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 check_osd_map already seen 605 >= 605 2013-04-03 10:39:13.308169 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 update_logger 2013-04-03 10:39:13.308180 7fcdffcdb700 0 log [INF] : pgmap v29038: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2621B/s wr, 0op/s 2013-04-03 10:39:13.308241 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos 2013-04-03 10:39:13.308259 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 2013-04-03 10:39:13.308278 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:13.308320 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos 2013-04-03 10:39:13.308334 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 2013-04-03 10:39:13.308343 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:13.308354 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:39:13.308386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:13.308399 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:39:13.308408 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 logging 2013-04-03 10:39:07.697912 mon.0 10.252.24.25:6789/0 103 : [INF] pgmap v29037: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:39:13.308429 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 logging 2013-04-03 10:39:13.308181 mon.0 10.252.24.25:6789/0 104 : [INF] pgmap v29038: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2621B/s wr, 0op/s 2013-04-03 10:39:13.333263 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos 2013-04-03 10:39:13.333279 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 2013-04-03 10:39:13.333294 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:13.346162 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29038 check_down_pgs 2013-04-03 10:39:13.346192 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29038 v29038: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2621B/s wr, 0op/s 2013-04-03 10:39:13.346218 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:39:13.346238 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:39:13.346274 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:39:13.346304 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos 2013-04-03 10:39:13.346322 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 2013-04-03 10:39:13.346330 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394039 log 2013-04-03 10:39:13.346346 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:13.346362 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:39:13.384518 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21742 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:39:14.307367 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394039 encode_pending v394040 2013-04-03 10:39:14.333702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos 2013-04-03 10:39:14.333728 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394039 2013-04-03 10:39:14.333744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos latest full 393798 2013-04-03 10:39:14.333763 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394040 update_from_paxos applying incremental log 394040 2013-04-03 10:39:07.697912 mon.0 10.252.24.25:6789/0 103 : [INF] pgmap v29037: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:39:14.333792 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394040 update_from_paxos applying incremental log 394040 2013-04-03 10:39:13.308181 mon.0 10.252.24.25:6789/0 104 : [INF] pgmap v29038: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2621B/s wr, 0op/s 2013-04-03 10:39:14.334037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 check_subs 2013-04-03 10:39:14.334074 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 create_pending v 394041 2013-04-03 10:39:14.334097 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394040 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:39:14.334163 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos 2013-04-03 10:39:14.334178 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 2013-04-03 10:39:14.334196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:14.347002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos 2013-04-03 10:39:14.347018 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 2013-04-03 10:39:14.347034 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:15.585321 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140372 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:16.931536 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:16.931546 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4000 from 10.252.24.88:6789/0 2013-04-03 10:39:16.931599 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 87 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:39:17.334629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos 2013-04-03 10:39:17.334649 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 2013-04-03 10:39:17.334666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:17.384690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21743 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:39:18.346511 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29038 v29038: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2621B/s wr, 0op/s 2013-04-03 10:39:18.346551 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:39:18.346574 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:39:18.346616 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:39:18.346647 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos 2013-04-03 10:39:18.346666 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 2013-04-03 10:39:18.346675 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394040 log 2013-04-03 10:39:18.346695 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:18.346714 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:39:19.585450 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140373 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:20.334710 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos 2013-04-03 10:39:20.334730 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 2013-04-03 10:39:20.334747 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:20.931595 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:20.931603 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52600 from 10.252.24.88:6789/0 2013-04-03 10:39:20.931636 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 88 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:39:21.384816 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21744 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:39:21.587878 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:21.587886 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:39:21.587920 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 preprocess_query pg_stats(1 pgs tid 29 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:21.587943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 prepare_update pg_stats(1 pgs tid 29 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:21.587949 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 prepare_pg_stats pg_stats(1 pgs tid 29 v 605) v1 from osd.1 2013-04-03 10:39:21.587955 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:39:21.638141 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29038 encode_pending v 29039 2013-04-03 10:39:21.689506 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29038 update_from_paxos applying incremental 29039 2013-04-03 10:39:21.689523 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 v29039: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1465B/s wr, 0op/s 2013-04-03 10:39:21.689543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 send_pg_creates to 0 pgs 2013-04-03 10:39:21.689548 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 update_logger 2013-04-03 10:39:21.689557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 create_pending v 29040 2013-04-03 10:39:21.689566 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29039 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:21.689581 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 check_osd_map already seen 605 >= 605 2013-04-03 10:39:21.689584 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 update_logger 2013-04-03 10:39:21.689595 7fcdffcdb700 0 log [INF] : pgmap v29039: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1465B/s wr, 0op/s 2013-04-03 10:39:21.689658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos 2013-04-03 10:39:21.689677 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 2013-04-03 10:39:21.689695 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:21.689737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos 2013-04-03 10:39:21.689752 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 2013-04-03 10:39:21.689761 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:21.689775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:39:21.689798 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:21.689810 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:39:21.689828 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 logging 2013-04-03 10:39:21.689596 mon.0 10.252.24.25:6789/0 105 : [INF] pgmap v29039: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1465B/s wr, 0op/s 2013-04-03 10:39:21.715162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos 2013-04-03 10:39:21.715179 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 2013-04-03 10:39:21.715193 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:22.689286 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394040 encode_pending v394041 2013-04-03 10:39:22.716760 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos 2013-04-03 10:39:22.716787 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394040 2013-04-03 10:39:22.716801 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos latest full 393798 2013-04-03 10:39:22.716819 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394041 update_from_paxos applying incremental log 394041 2013-04-03 10:39:21.689596 mon.0 10.252.24.25:6789/0 105 : [INF] pgmap v29039: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1465B/s wr, 0op/s 2013-04-03 10:39:22.716866 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 check_subs 2013-04-03 10:39:22.716898 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 create_pending v 394042 2013-04-03 10:39:22.716920 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394041 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:39:22.716986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos 2013-04-03 10:39:22.717002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 2013-04-03 10:39:22.717020 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:22.728644 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos 2013-04-03 10:39:22.728660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 2013-04-03 10:39:22.728675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:23.346836 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29039 check_down_pgs 2013-04-03 10:39:23.346865 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29039 v29039: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1465B/s wr, 0op/s 2013-04-03 10:39:23.346892 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:39:23.346916 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:39:23.346953 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:39:23.346984 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos 2013-04-03 10:39:23.347002 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 2013-04-03 10:39:23.347012 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394041 log 2013-04-03 10:39:23.347029 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:23.347048 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:39:23.585547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140374 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:24.931766 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:24.931776 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x295e340 from 10.252.24.88:6789/0 2013-04-03 10:39:24.931828 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 89 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:39:25.384942 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21745 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:39:25.717633 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos 2013-04-03 10:39:25.717652 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 2013-04-03 10:39:25.717669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:27.585740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140375 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:28.227953 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:28.227963 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:39:28.228013 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 preprocess_query pg_stats(1 pgs tid 10 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:39:28.228037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 prepare_update pg_stats(1 pgs tid 10 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:39:28.228043 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 prepare_pg_stats pg_stats(1 pgs tid 10 v 605) v1 from osd.2 2013-04-03 10:39:28.228050 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:39:28.278205 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29039 encode_pending v 29040 2013-04-03 10:39:28.321460 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29039 update_from_paxos applying incremental 29040 2013-04-03 10:39:28.321476 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29040 v29040: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:39:28.321494 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29040 send_pg_creates to 0 pgs 2013-04-03 10:39:28.321497 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29040 update_logger 2013-04-03 10:39:28.321506 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29040 create_pending v 29041 2013-04-03 10:39:28.321514 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29040 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:39:28.321530 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29040 check_osd_map already seen 605 >= 605 2013-04-03 10:39:28.321533 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29040 update_logger 2013-04-03 10:39:28.321543 7fcdffcdb700 0 log [INF] : pgmap v29040: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:39:28.321603 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos 2013-04-03 10:39:28.321622 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 2013-04-03 10:39:28.321640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:28.321683 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos 2013-04-03 10:39:28.321699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 2013-04-03 10:39:28.321708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:28.321721 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:39:28.321743 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:28.321755 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:39:28.321766 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 logging 2013-04-03 10:39:28.321544 mon.0 10.252.24.25:6789/0 106 : [INF] pgmap v29040: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:39:28.347163 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29040 check_down_pgs 2013-04-03 10:39:28.347199 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29040 v29040: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:39:28.347222 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:39:28.347243 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:39:28.347279 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:39:28.347309 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos 2013-04-03 10:39:28.347325 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 2013-04-03 10:39:28.347334 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394041 log 2013-04-03 10:39:28.347350 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:28.347366 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:39:28.349611 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos 2013-04-03 10:39:28.349627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 2013-04-03 10:39:28.349643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:39:28.485071 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:39:28.485076 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:39:28.485083 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 2013-04-03 10:39:28.931832 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:28.931841 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83580 from 10.252.24.88:6789/0 2013-04-03 10:39:28.931879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 90 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:39:29.307047 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394041 encode_pending v394042 2013-04-03 10:39:29.338680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos 2013-04-03 10:39:29.338707 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos version 394042 summary v 394041 2013-04-03 10:39:29.338722 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos latest full 393798 2013-04-03 10:39:29.338742 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos applying incremental log 394042 2013-04-03 10:39:28.321544 mon.0 10.252.24.25:6789/0 106 : [INF] pgmap v29040: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:39:29.338789 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 check_subs 2013-04-03 10:39:29.338821 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 create_pending v 394043 2013-04-03 10:39:29.338844 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:39:29.338909 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos 2013-04-03 10:39:29.338925 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos version 394042 summary v 394042 2013-04-03 10:39:29.338944 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:29.353692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos 2013-04-03 10:39:29.353709 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos version 394042 summary v 394042 2013-04-03 10:39:29.353725 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:29.385013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21746 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:39:31.585859 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140376 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:31.588953 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:31.588962 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:39:31.588999 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29040 preprocess_query pg_stats(1 pgs tid 30 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:31.589024 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29040 prepare_update pg_stats(1 pgs tid 30 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:31.589030 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29040 prepare_pg_stats pg_stats(1 pgs tid 30 v 605) v1 from osd.1 2013-04-03 10:39:31.589037 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29040 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:39:31.639218 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29040 encode_pending v 29041 2013-04-03 10:39:31.695153 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29040 update_from_paxos applying incremental 29041 2013-04-03 10:39:31.695170 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 v29041: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 4197B/s wr, 0op/s 2013-04-03 10:39:31.695193 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 send_pg_creates to 0 pgs 2013-04-03 10:39:31.695198 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 update_logger 2013-04-03 10:39:31.695208 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 create_pending v 29042 2013-04-03 10:39:31.695217 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:31.695233 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 check_osd_map already seen 605 >= 605 2013-04-03 10:39:31.695237 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 update_logger 2013-04-03 10:39:31.695248 7fcdffcdb700 0 log [INF] : pgmap v29041: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 4197B/s wr, 0op/s 2013-04-03 10:39:31.695305 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos 2013-04-03 10:39:31.695324 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos version 394042 summary v 394042 2013-04-03 10:39:31.695344 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:31.695387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos 2013-04-03 10:39:31.695405 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos version 394042 summary v 394042 2013-04-03 10:39:31.695414 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:31.695427 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:39:31.695451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:31.695463 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:39:31.695473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 logging 2013-04-03 10:39:31.695249 mon.0 10.252.24.25:6789/0 107 : [INF] pgmap v29041: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 4197B/s wr, 0op/s 2013-04-03 10:39:31.707933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos 2013-04-03 10:39:31.707951 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos version 394042 summary v 394042 2013-04-03 10:39:31.707967 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:31.754354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:31.754362 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:39:31.754392 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 preprocess_query pg_stats(2 pgs tid 30446 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:39:31.754413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 prepare_update pg_stats(2 pgs tid 30446 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:39:31.754418 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 prepare_pg_stats pg_stats(2 pgs tid 30446 v 605) v1 from osd.3 2013-04-03 10:39:31.754422 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 got osd.3 osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:39:32.694411 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 encode_pending v 29042 2013-04-03 10:39:32.695655 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 encode_pending v394043 2013-04-03 10:39:32.725052 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 update_from_paxos applying incremental 29042 2013-04-03 10:39:32.725075 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 v29042: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 79532B/s wr, 1op/s 2013-04-03 10:39:32.725096 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 send_pg_creates to 0 pgs 2013-04-03 10:39:32.725100 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 update_logger 2013-04-03 10:39:32.725114 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 create_pending v 29043 2013-04-03 10:39:32.725125 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:39:32.725143 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 check_osd_map already seen 605 >= 605 2013-04-03 10:39:32.725146 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 update_logger 2013-04-03 10:39:32.725157 7fcdffcdb700 0 log [INF] : pgmap v29042: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 79532B/s wr, 1op/s 2013-04-03 10:39:32.901577 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos 2013-04-03 10:39:32.901604 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394042 2013-04-03 10:39:32.901618 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos latest full 393798 2013-04-03 10:39:32.901637 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos applying incremental log 394043 2013-04-03 10:39:31.695249 mon.0 10.252.24.25:6789/0 107 : [INF] pgmap v29041: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 4197B/s wr, 0op/s 2013-04-03 10:39:32.901684 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 check_subs 2013-04-03 10:39:32.901717 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 create_pending v 394044 2013-04-03 10:39:32.901740 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:39:32.901773 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos 2013-04-03 10:39:32.901789 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394043 2013-04-03 10:39:32.901798 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:32.901818 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:39:32.901839 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 nothing new 2013-04-03 10:39:32.901889 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos 2013-04-03 10:39:32.901905 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394043 2013-04-03 10:39:32.901923 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:33.018354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos 2013-04-03 10:39:33.018371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394043 2013-04-03 10:39:33.018387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:33.018666 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:33.018674 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a84b80 from 10.252.24.88:6789/0 2013-04-03 10:39:33.018700 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 91 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:39:33.347509 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 check_down_pgs 2013-04-03 10:39:33.347545 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 v29042: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 79532B/s wr, 1op/s 2013-04-03 10:39:33.347573 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:39:33.347601 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:39:33.347639 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:39:33.347671 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos 2013-04-03 10:39:33.347690 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394043 2013-04-03 10:39:33.347700 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 log 2013-04-03 10:39:33.347718 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:33.347738 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:39:33.385149 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21747 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:39:35.462686 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:35.462696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:39:35.462749 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 preprocess_query pg_stats(0 pgs tid 60 v 605) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:39:35.462778 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 prepare_update pg_stats(0 pgs tid 60 v 605) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:39:35.462784 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 prepare_pg_stats pg_stats(0 pgs tid 60 v 605) v1 from osd.4 2013-04-03 10:39:35.462791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 got osd.4 osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[])) 2013-04-03 10:39:35.512968 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 encode_pending v 29043 2013-04-03 10:39:35.537516 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 update_from_paxos applying incremental 29043 2013-04-03 10:39:35.537527 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 v29043: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 83003B/s wr, 0op/s 2013-04-03 10:39:35.537545 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 send_pg_creates to 0 pgs 2013-04-03 10:39:35.537549 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 update_logger 2013-04-03 10:39:35.537559 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 create_pending v 29044 2013-04-03 10:39:35.537569 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 _updated_stats for osd.4 10.252.24.32:6800/13473 2013-04-03 10:39:35.537584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 check_osd_map already seen 605 >= 605 2013-04-03 10:39:35.537588 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 update_logger 2013-04-03 10:39:35.537598 7fcdffcdb700 0 log [INF] : pgmap v29043: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 83003B/s wr, 0op/s 2013-04-03 10:39:35.537672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos 2013-04-03 10:39:35.537692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394043 2013-04-03 10:39:35.537712 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:35.537755 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos 2013-04-03 10:39:35.537770 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394043 2013-04-03 10:39:35.537779 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:35.537791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:39:35.537831 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:35.537846 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:39:35.537858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 logging 2013-04-03 10:39:32.725158 mon.0 10.252.24.25:6789/0 108 : [INF] pgmap v29042: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 79532B/s wr, 1op/s 2013-04-03 10:39:35.537881 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 logging 2013-04-03 10:39:35.537600 mon.0 10.252.24.25:6789/0 109 : [INF] pgmap v29043: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 83003B/s wr, 0op/s 2013-04-03 10:39:35.550124 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos 2013-04-03 10:39:35.550140 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394043 2013-04-03 10:39:35.550158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:35.585929 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140377 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:36.536579 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 encode_pending v394044 2013-04-03 10:39:36.575565 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos 2013-04-03 10:39:36.575593 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos version 394044 summary v 394043 2013-04-03 10:39:36.575607 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos latest full 393798 2013-04-03 10:39:36.575625 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos applying incremental log 394044 2013-04-03 10:39:32.725158 mon.0 10.252.24.25:6789/0 108 : [INF] pgmap v29042: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 79532B/s wr, 1op/s 2013-04-03 10:39:36.575654 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos applying incremental log 394044 2013-04-03 10:39:35.537600 mon.0 10.252.24.25:6789/0 109 : [INF] pgmap v29043: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 83003B/s wr, 0op/s 2013-04-03 10:39:36.575694 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 check_subs 2013-04-03 10:39:36.575726 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 create_pending v 394045 2013-04-03 10:39:36.575749 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:39:36.575814 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos 2013-04-03 10:39:36.575831 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos version 394044 summary v 394044 2013-04-03 10:39:36.575849 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:36.587592 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos 2013-04-03 10:39:36.587609 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos version 394044 summary v 394044 2013-04-03 10:39:36.587626 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:36.589519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:36.589528 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:39:36.589557 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 preprocess_query pg_stats(1 pgs tid 31 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:36.589579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 prepare_update pg_stats(1 pgs tid 31 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:36.589585 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 prepare_pg_stats pg_stats(1 pgs tid 31 v 605) v1 from osd.1 2013-04-03 10:39:36.589592 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:39:36.818138 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:36.818147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:39:36.818178 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 preprocess_query pg_stats(0 pgs tid 409 v 605) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:39:36.818199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 prepare_update pg_stats(0 pgs tid 409 v 605) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:39:36.818205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 prepare_pg_stats pg_stats(0 pgs tid 409 v 605) v1 from osd.0 2013-04-03 10:39:36.818211 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 got osd.0 osd_stat(1139 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1139 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:39:36.932118 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:36.932126 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83000 from 10.252.24.88:6789/0 2013-04-03 10:39:36.932157 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 92 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:39:37.385286 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21748 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:39:37.575348 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 encode_pending v 29044 2013-04-03 10:39:37.613632 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 update_from_paxos applying incremental 29044 2013-04-03 10:39:37.613654 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 v29044: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s 2013-04-03 10:39:37.613674 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 send_pg_creates to 0 pgs 2013-04-03 10:39:37.613679 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 update_logger 2013-04-03 10:39:37.613690 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 create_pending v 29045 2013-04-03 10:39:37.613699 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:37.613719 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:39:37.613758 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 check_osd_map already seen 605 >= 605 2013-04-03 10:39:37.613763 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 update_logger 2013-04-03 10:39:37.613774 7fcdffcdb700 0 log [INF] : pgmap v29044: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s 2013-04-03 10:39:37.613843 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos 2013-04-03 10:39:37.613863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos version 394044 summary v 394044 2013-04-03 10:39:37.613882 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:37.613926 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos 2013-04-03 10:39:37.613941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos version 394044 summary v 394044 2013-04-03 10:39:37.613950 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:37.613962 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:39:37.613985 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:37.613998 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:39:37.614013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 logging 2013-04-03 10:39:37.613775 mon.0 10.252.24.25:6789/0 110 : [INF] pgmap v29044: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s 2013-04-03 10:39:37.626317 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos 2013-04-03 10:39:37.626333 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos version 394044 summary v 394044 2013-04-03 10:39:37.626348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:38.347887 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 check_down_pgs 2013-04-03 10:39:38.347917 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 v29044: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s 2013-04-03 10:39:38.347946 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:39:38.347971 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:39:38.348008 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:39:38.348040 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos 2013-04-03 10:39:38.348064 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos version 394044 summary v 394044 2013-04-03 10:39:38.348074 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 log 2013-04-03 10:39:38.348091 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:38.348110 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:39:38.612805 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 encode_pending v394045 2013-04-03 10:39:38.646147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos 2013-04-03 10:39:38.646174 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos version 394045 summary v 394044 2013-04-03 10:39:38.646189 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos latest full 393798 2013-04-03 10:39:38.646208 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos applying incremental log 394045 2013-04-03 10:39:37.613775 mon.0 10.252.24.25:6789/0 110 : [INF] pgmap v29044: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s 2013-04-03 10:39:38.646254 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 check_subs 2013-04-03 10:39:38.646287 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 create_pending v 394046 2013-04-03 10:39:38.646310 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:39:38.646369 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos 2013-04-03 10:39:38.646385 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos version 394045 summary v 394045 2013-04-03 10:39:38.646402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:38.660247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos 2013-04-03 10:39:38.660264 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos version 394045 summary v 394045 2013-04-03 10:39:38.660281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:39.586101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140378 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:40.932277 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:40.932288 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a832c0 from 10.252.24.88:6789/0 2013-04-03 10:39:40.932340 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 93 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:39:41.385382 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21749 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:39:41.590067 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:41.590076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:39:41.590111 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 preprocess_query pg_stats(1 pgs tid 32 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:41.590136 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 prepare_update pg_stats(1 pgs tid 32 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:41.590142 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 prepare_pg_stats pg_stats(1 pgs tid 32 v 605) v1 from osd.1 2013-04-03 10:39:41.590149 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:39:41.640333 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 encode_pending v 29045 2013-04-03 10:39:41.774936 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 update_from_paxos applying incremental 29045 2013-04-03 10:39:41.774953 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 v29045: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3509B/s wr, 0op/s 2013-04-03 10:39:41.774972 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 send_pg_creates to 0 pgs 2013-04-03 10:39:41.774977 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 update_logger 2013-04-03 10:39:41.774987 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 create_pending v 29046 2013-04-03 10:39:41.774996 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:41.775021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 check_osd_map already seen 605 >= 605 2013-04-03 10:39:41.775025 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 update_logger 2013-04-03 10:39:41.775036 7fcdffcdb700 0 log [INF] : pgmap v29045: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3509B/s wr, 0op/s 2013-04-03 10:39:41.775097 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos 2013-04-03 10:39:41.775116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos version 394045 summary v 394045 2013-04-03 10:39:41.775135 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:41.775197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos 2013-04-03 10:39:41.775213 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos version 394045 summary v 394045 2013-04-03 10:39:41.775222 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:41.775233 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:39:41.775258 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:41.775271 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:39:41.775282 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 logging 2013-04-03 10:39:41.775037 mon.0 10.252.24.25:6789/0 111 : [INF] pgmap v29045: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3509B/s wr, 0op/s 2013-04-03 10:39:41.786256 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos 2013-04-03 10:39:41.786273 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos version 394045 summary v 394045 2013-04-03 10:39:41.786288 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:42.774680 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 encode_pending v394046 2013-04-03 10:39:42.819281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos 2013-04-03 10:39:42.819309 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos version 394046 summary v 394045 2013-04-03 10:39:42.819323 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos latest full 393798 2013-04-03 10:39:42.819342 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos applying incremental log 394046 2013-04-03 10:39:41.775037 mon.0 10.252.24.25:6789/0 111 : [INF] pgmap v29045: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3509B/s wr, 0op/s 2013-04-03 10:39:42.819390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 check_subs 2013-04-03 10:39:42.819421 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 create_pending v 394047 2013-04-03 10:39:42.819444 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:39:42.819509 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos 2013-04-03 10:39:42.819525 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos version 394046 summary v 394046 2013-04-03 10:39:42.819885 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:42.830908 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos 2013-04-03 10:39:42.830925 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos version 394046 summary v 394046 2013-04-03 10:39:42.830944 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:43.230118 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:43.230130 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:39:43.230182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 preprocess_query pg_stats(1 pgs tid 11 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:39:43.230206 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 prepare_update pg_stats(1 pgs tid 11 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:39:43.230213 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 prepare_pg_stats pg_stats(1 pgs tid 11 v 605) v1 from osd.2 2013-04-03 10:39:43.230220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:39:43.348258 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 check_down_pgs 2013-04-03 10:39:43.348283 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 v29045: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3509B/s wr, 0op/s 2013-04-03 10:39:43.348311 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:39:43.348337 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:39:43.348374 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:39:43.348406 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos 2013-04-03 10:39:43.348425 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos version 394046 summary v 394046 2013-04-03 10:39:43.348434 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 log 2013-04-03 10:39:43.348451 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:43.348470 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:39:43.586186 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140379 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:43.819104 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 encode_pending v 29046 2013-04-03 10:39:43.847461 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 update_from_paxos applying incremental 29046 2013-04-03 10:39:43.847480 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 v29046: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3936B/s wr, 0op/s 2013-04-03 10:39:43.847498 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 send_pg_creates to 0 pgs 2013-04-03 10:39:43.847503 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 update_logger 2013-04-03 10:39:43.847513 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 create_pending v 29047 2013-04-03 10:39:43.847527 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:39:43.847545 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 check_osd_map already seen 605 >= 605 2013-04-03 10:39:43.847548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 update_logger 2013-04-03 10:39:43.847560 7fcdffcdb700 0 log [INF] : pgmap v29046: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3936B/s wr, 0op/s 2013-04-03 10:39:43.847616 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos 2013-04-03 10:39:43.847634 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos version 394046 summary v 394046 2013-04-03 10:39:43.847652 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:43.847695 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos 2013-04-03 10:39:43.847710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos version 394046 summary v 394046 2013-04-03 10:39:43.847718 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:43.847729 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:39:43.847753 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:43.847766 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:39:43.847781 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 logging 2013-04-03 10:39:43.847561 mon.0 10.252.24.25:6789/0 112 : [INF] pgmap v29046: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3936B/s wr, 0op/s 2013-04-03 10:39:43.872002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos 2013-04-03 10:39:43.872019 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos version 394046 summary v 394046 2013-04-03 10:39:43.872035 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:44.847287 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 encode_pending v394047 2013-04-03 10:39:44.871575 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos 2013-04-03 10:39:44.871610 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394046 2013-04-03 10:39:44.871625 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos latest full 393798 2013-04-03 10:39:44.871645 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos applying incremental log 394047 2013-04-03 10:39:43.847561 mon.0 10.252.24.25:6789/0 112 : [INF] pgmap v29046: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3936B/s wr, 0op/s 2013-04-03 10:39:44.871692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 check_subs 2013-04-03 10:39:44.871724 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 create_pending v 394048 2013-04-03 10:39:44.871747 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:39:44.871811 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos 2013-04-03 10:39:44.871827 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394047 2013-04-03 10:39:44.871846 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:44.883188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos 2013-04-03 10:39:44.883205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394047 2013-04-03 10:39:44.883220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:44.932473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:44.932482 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2996b80 from 10.252.24.88:6789/0 2013-04-03 10:39:44.932512 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 94 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:39:45.385500 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21750 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:39:47.586362 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140380 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:47.872413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos 2013-04-03 10:39:47.872433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394047 2013-04-03 10:39:47.872451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:48.231028 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:48.231040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:39:48.231092 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 preprocess_query pg_stats(1 pgs tid 12 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:39:48.231117 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 prepare_update pg_stats(1 pgs tid 12 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:39:48.231123 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 prepare_pg_stats pg_stats(1 pgs tid 12 v 605) v1 from osd.2 2013-04-03 10:39:48.231130 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:39:48.281313 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 encode_pending v 29047 2013-04-03 10:39:48.313875 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 update_from_paxos applying incremental 29047 2013-04-03 10:39:48.313897 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 v29047: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2467B/s wr, 0op/s 2013-04-03 10:39:48.313916 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 send_pg_creates to 0 pgs 2013-04-03 10:39:48.313919 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 update_logger 2013-04-03 10:39:48.313929 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 create_pending v 29048 2013-04-03 10:39:48.313939 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:39:48.313962 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 check_osd_map already seen 605 >= 605 2013-04-03 10:39:48.313965 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 update_logger 2013-04-03 10:39:48.313976 7fcdffcdb700 0 log [INF] : pgmap v29047: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2467B/s wr, 0op/s 2013-04-03 10:39:48.314038 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos 2013-04-03 10:39:48.314058 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394047 2013-04-03 10:39:48.314076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:48.314119 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos 2013-04-03 10:39:48.314134 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394047 2013-04-03 10:39:48.314144 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:48.314155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:39:48.314178 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:48.314190 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:39:48.314201 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 logging 2013-04-03 10:39:48.313977 mon.0 10.252.24.25:6789/0 113 : [INF] pgmap v29047: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2467B/s wr, 0op/s 2013-04-03 10:39:48.338612 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos 2013-04-03 10:39:48.338630 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394047 2013-04-03 10:39:48.338646 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:48.348582 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 check_down_pgs 2013-04-03 10:39:48.348611 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 v29047: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2467B/s wr, 0op/s 2013-04-03 10:39:48.348636 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:39:48.348658 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:39:48.348694 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:39:48.348725 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos 2013-04-03 10:39:48.348742 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394047 2013-04-03 10:39:48.348751 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 log 2013-04-03 10:39:48.348767 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:48.348783 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:39:48.932489 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:48.932498 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2995580 from 10.252.24.88:6789/0 2013-04-03 10:39:48.932533 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 95 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:39:49.313583 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 encode_pending v394048 2013-04-03 10:39:49.338296 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos 2013-04-03 10:39:49.338325 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos version 394048 summary v 394047 2013-04-03 10:39:49.338478 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos latest full 393798 2013-04-03 10:39:49.338502 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos applying incremental log 394048 2013-04-03 10:39:48.313977 mon.0 10.252.24.25:6789/0 113 : [INF] pgmap v29047: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2467B/s wr, 0op/s 2013-04-03 10:39:49.338547 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 check_subs 2013-04-03 10:39:49.338578 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 create_pending v 394049 2013-04-03 10:39:49.338601 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:39:49.338671 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos 2013-04-03 10:39:49.338687 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos version 394048 summary v 394048 2013-04-03 10:39:49.338706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:49.349935 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos 2013-04-03 10:39:49.349951 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos version 394048 summary v 394048 2013-04-03 10:39:49.349968 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:49.385543 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21751 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:39:51.586481 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140381 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:51.591155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:51.591166 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:39:51.591199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 preprocess_query pg_stats(1 pgs tid 33 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:51.591223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 prepare_update pg_stats(1 pgs tid 33 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:51.591230 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 prepare_pg_stats pg_stats(1 pgs tid 33 v 605) v1 from osd.1 2013-04-03 10:39:51.591236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:39:51.641418 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 encode_pending v 29048 2013-04-03 10:39:51.807156 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 update_from_paxos applying incremental 29048 2013-04-03 10:39:51.807172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 v29048: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1309B/s wr, 0op/s 2013-04-03 10:39:51.807192 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 send_pg_creates to 0 pgs 2013-04-03 10:39:51.807197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 update_logger 2013-04-03 10:39:51.807207 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 create_pending v 29049 2013-04-03 10:39:51.807217 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:39:51.807233 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 check_osd_map already seen 605 >= 605 2013-04-03 10:39:51.807236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 update_logger 2013-04-03 10:39:51.807247 7fcdffcdb700 0 log [INF] : pgmap v29048: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1309B/s wr, 0op/s 2013-04-03 10:39:51.807308 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos 2013-04-03 10:39:51.807328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos version 394048 summary v 394048 2013-04-03 10:39:51.807347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:51.807390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos 2013-04-03 10:39:51.807405 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos version 394048 summary v 394048 2013-04-03 10:39:51.807415 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:51.807427 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:39:51.807450 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:51.807463 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:39:51.807473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 logging 2013-04-03 10:39:51.807248 mon.0 10.252.24.25:6789/0 114 : [INF] pgmap v29048: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1309B/s wr, 0op/s 2013-04-03 10:39:51.831302 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos 2013-04-03 10:39:51.831319 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos version 394048 summary v 394048 2013-04-03 10:39:51.831334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:51.831592 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:51.831599 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:39:51.831625 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 preprocess_query pg_stats(1 pgs tid 30447 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:39:51.831645 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 prepare_update pg_stats(1 pgs tid 30447 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:39:51.831651 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 prepare_pg_stats pg_stats(1 pgs tid 30447 v 605) v1 from osd.3 2013-04-03 10:39:51.831656 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:39:52.806289 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 encode_pending v394049 2013-04-03 10:39:52.808363 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 encode_pending v 29049 2013-04-03 10:39:52.839088 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos 2013-04-03 10:39:52.839116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos version 394049 summary v 394048 2013-04-03 10:39:52.839133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos latest full 393798 2013-04-03 10:39:52.839152 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos applying incremental log 394049 2013-04-03 10:39:51.807248 mon.0 10.252.24.25:6789/0 114 : [INF] pgmap v29048: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1309B/s wr, 0op/s 2013-04-03 10:39:52.839202 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 check_subs 2013-04-03 10:39:52.839234 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 create_pending v 394050 2013-04-03 10:39:52.839257 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:39:52.864224 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 update_from_paxos applying incremental 29049 2013-04-03 10:39:52.864243 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s 2013-04-03 10:39:52.864263 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 send_pg_creates to 0 pgs 2013-04-03 10:39:52.864268 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 update_logger 2013-04-03 10:39:52.864278 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 create_pending v 29050 2013-04-03 10:39:52.864287 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:39:52.864304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 check_osd_map already seen 605 >= 605 2013-04-03 10:39:52.864307 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 update_logger 2013-04-03 10:39:52.864317 7fcdffcdb700 0 log [INF] : pgmap v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s 2013-04-03 10:39:52.864382 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos 2013-04-03 10:39:52.864399 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos version 394049 summary v 394049 2013-04-03 10:39:52.864418 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:52.864460 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos 2013-04-03 10:39:52.864476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos version 394049 summary v 394049 2013-04-03 10:39:52.864485 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:52.864497 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:39:52.864517 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:39:52.864530 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:39:52.864540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 logging 2013-04-03 10:39:52.864318 mon.0 10.252.24.25:6789/0 115 : [INF] pgmap v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s 2013-04-03 10:39:52.889187 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos 2013-04-03 10:39:52.889204 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos version 394049 summary v 394049 2013-04-03 10:39:52.889221 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:52.932500 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:52.932510 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ce2600 from 10.252.24.88:6789/0 2013-04-03 10:39:52.932540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 96 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:39:53.348900 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 check_down_pgs 2013-04-03 10:39:53.348931 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s 2013-04-03 10:39:53.348959 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:39:53.348985 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:39:53.349023 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:39:53.349054 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos 2013-04-03 10:39:53.349074 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos version 394049 summary v 394049 2013-04-03 10:39:53.349083 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 log 2013-04-03 10:39:53.349100 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:53.349120 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:39:53.385724 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21752 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:39:53.864026 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 encode_pending v394050 2013-04-03 10:39:53.979171 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos 2013-04-03 10:39:53.979198 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos version 394050 summary v 394049 2013-04-03 10:39:53.979213 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos latest full 393798 2013-04-03 10:39:53.979233 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos applying incremental log 394050 2013-04-03 10:39:52.864318 mon.0 10.252.24.25:6789/0 115 : [INF] pgmap v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s 2013-04-03 10:39:53.979278 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 check_subs 2013-04-03 10:39:53.979309 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 create_pending v 394051 2013-04-03 10:39:53.979332 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:39:53.979395 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos 2013-04-03 10:39:53.979411 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos version 394050 summary v 394050 2013-04-03 10:39:53.979433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:53.992022 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos 2013-04-03 10:39:53.992039 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos version 394050 summary v 394050 2013-04-03 10:39:53.992057 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:55.586594 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140382 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:56.932782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:39:56.932792 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ce28c0 from 10.252.24.88:6789/0 2013-04-03 10:39:56.932844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 97 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:39:56.979986 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos 2013-04-03 10:39:56.980007 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos version 394050 summary v 394050 2013-04-03 10:39:56.980028 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:57.385775 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21753 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:39:58.349263 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s 2013-04-03 10:39:58.349309 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:39:58.349341 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:39:58.349388 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:39:58.349419 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos 2013-04-03 10:39:58.349439 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos version 394050 summary v 394050 2013-04-03 10:39:58.349576 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 log 2013-04-03 10:39:58.349600 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:39:58.349620 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:39:58.485153 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) 2013-04-03 10:39:58.485198 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader 2013-04-03 10:39:59.586738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140383 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:39:59.980212 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos 2013-04-03 10:39:59.980231 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 2013-04-03 10:39:59.980248 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:00.633218 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33075/0) 2013-04-03 10:40:00.932853 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:00.932862 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc5600 from 10.252.24.88:6789/0 2013-04-03 10:40:00.932914 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 98 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:40:01.385905 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21754 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:40:02.980138 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos 2013-04-03 10:40:02.980158 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 2013-04-03 10:40:02.980175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:03.349759 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29049 v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s 2013-04-03 10:40:03.349802 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:40:03.349824 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:40:03.349873 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:40:03.349903 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos 2013-04-03 10:40:03.349922 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 2013-04-03 10:40:03.349931 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394050 log 2013-04-03 10:40:03.349948 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:03.349966 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:40:03.586838 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140384 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:40:04.932975 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:04.932985 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4580 from 10.252.24.88:6789/0 2013-04-03 10:40:04.933038 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 99 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:40:05.385982 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21755 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:40:05.980256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos 2013-04-03 10:40:05.980275 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 2013-04-03 10:40:05.980293 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:07.587016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140385 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:40:08.233458 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:08.233468 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:40:08.233518 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29049 preprocess_query pg_stats(1 pgs tid 13 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:40:08.233542 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29049 prepare_update pg_stats(1 pgs tid 13 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:40:08.233548 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29049 prepare_pg_stats pg_stats(1 pgs tid 13 v 605) v1 from osd.2 2013-04-03 10:40:08.233555 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29049 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:40:08.283750 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29049 encode_pending v 29050 2013-04-03 10:40:08.320973 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29049 update_from_paxos applying incremental 29050 2013-04-03 10:40:08.320996 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 v29050: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 43870B/s wr, 0op/s 2013-04-03 10:40:08.321016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 send_pg_creates to 0 pgs 2013-04-03 10:40:08.321019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 update_logger 2013-04-03 10:40:08.321029 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 create_pending v 29051 2013-04-03 10:40:08.321037 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29050 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:40:08.321054 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 check_osd_map already seen 605 >= 605 2013-04-03 10:40:08.321057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 update_logger 2013-04-03 10:40:08.321067 7fcdffcdb700 0 log [INF] : pgmap v29050: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 43870B/s wr, 0op/s 2013-04-03 10:40:08.321136 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos 2013-04-03 10:40:08.321155 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 2013-04-03 10:40:08.321173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:08.321216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos 2013-04-03 10:40:08.321231 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 2013-04-03 10:40:08.321240 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:08.321253 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:40:08.321278 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:08.321290 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:40:08.321301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 logging 2013-04-03 10:40:08.321068 mon.0 10.252.24.25:6789/0 116 : [INF] pgmap v29050: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 43870B/s wr, 0op/s 2013-04-03 10:40:08.331974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos 2013-04-03 10:40:08.331990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 2013-04-03 10:40:08.332006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:08.350082 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29050 check_down_pgs 2013-04-03 10:40:08.350118 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29050 v29050: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 43870B/s wr, 0op/s 2013-04-03 10:40:08.350141 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:40:08.350165 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:40:08.350201 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:40:08.350231 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos 2013-04-03 10:40:08.350248 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 2013-04-03 10:40:08.350257 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394050 log 2013-04-03 10:40:08.350275 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:08.350291 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:40:08.933251 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:08.933261 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc5080 from 10.252.24.88:6789/0 2013-04-03 10:40:08.933312 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 100 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:40:09.320107 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394050 encode_pending v394051 2013-04-03 10:40:09.348383 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos 2013-04-03 10:40:09.348409 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos version 394051 summary v 394050 2013-04-03 10:40:09.348423 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos latest full 393798 2013-04-03 10:40:09.348444 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394051 update_from_paxos applying incremental log 394051 2013-04-03 10:40:08.321068 mon.0 10.252.24.25:6789/0 116 : [INF] pgmap v29050: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 43870B/s wr, 0op/s 2013-04-03 10:40:09.348489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 check_subs 2013-04-03 10:40:09.348520 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 create_pending v 394052 2013-04-03 10:40:09.348543 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394051 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:40:09.348607 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos 2013-04-03 10:40:09.348623 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos version 394051 summary v 394051 2013-04-03 10:40:09.348641 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:09.359856 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos 2013-04-03 10:40:09.359872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos version 394051 summary v 394051 2013-04-03 10:40:09.359888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:09.386055 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21756 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:40:10.649208 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33076/0) 2013-04-03 10:40:11.587127 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140386 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:40:11.758972 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:11.758981 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:40:11.759014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 preprocess_query pg_stats(1 pgs tid 30448 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:40:11.759038 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 prepare_update pg_stats(1 pgs tid 30448 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:40:11.759043 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 prepare_pg_stats pg_stats(1 pgs tid 30448 v 605) v1 from osd.3 2013-04-03 10:40:11.759049 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:40:11.809230 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29050 encode_pending v 29051 2013-04-03 10:40:11.842000 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29050 update_from_paxos applying incremental 29051 2013-04-03 10:40:11.842015 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 v29051: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 38371B/s wr, 0op/s 2013-04-03 10:40:11.842034 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 send_pg_creates to 0 pgs 2013-04-03 10:40:11.842041 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 update_logger 2013-04-03 10:40:11.842051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 create_pending v 29052 2013-04-03 10:40:11.842060 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29051 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:40:11.842075 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 check_osd_map already seen 605 >= 605 2013-04-03 10:40:11.842077 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 update_logger 2013-04-03 10:40:11.842088 7fcdffcdb700 0 log [INF] : pgmap v29051: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 38371B/s wr, 0op/s 2013-04-03 10:40:11.842143 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos 2013-04-03 10:40:11.842162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos version 394051 summary v 394051 2013-04-03 10:40:11.842180 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:11.842222 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos 2013-04-03 10:40:11.842237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos version 394051 summary v 394051 2013-04-03 10:40:11.842246 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:11.842258 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:40:11.842285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:11.842297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:40:11.842308 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 logging 2013-04-03 10:40:11.842089 mon.0 10.252.24.25:6789/0 117 : [INF] pgmap v29051: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 38371B/s wr, 0op/s 2013-04-03 10:40:11.853481 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos 2013-04-03 10:40:11.853498 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos version 394051 summary v 394051 2013-04-03 10:40:11.853514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:12.841761 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394051 encode_pending v394052 2013-04-03 10:40:12.872019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos 2013-04-03 10:40:12.872044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394051 2013-04-03 10:40:12.872062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos latest full 393798 2013-04-03 10:40:12.872081 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394052 update_from_paxos applying incremental log 394052 2013-04-03 10:40:11.842089 mon.0 10.252.24.25:6789/0 117 : [INF] pgmap v29051: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 38371B/s wr, 0op/s 2013-04-03 10:40:12.872125 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 check_subs 2013-04-03 10:40:12.872156 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 create_pending v 394053 2013-04-03 10:40:12.872178 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394052 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:40:12.872242 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos 2013-04-03 10:40:12.872260 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 2013-04-03 10:40:12.872279 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:12.896362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos 2013-04-03 10:40:12.896378 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 2013-04-03 10:40:12.896394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:12.933181 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:12.933189 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c5b80 from 10.252.24.88:6789/0 2013-04-03 10:40:12.933224 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 101 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:40:13.350433 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29051 check_down_pgs 2013-04-03 10:40:13.350468 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29051 v29051: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 38371B/s wr, 0op/s 2013-04-03 10:40:13.350496 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:40:13.350520 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:40:13.350556 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:40:13.350592 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos 2013-04-03 10:40:13.350611 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 2013-04-03 10:40:13.350620 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394052 log 2013-04-03 10:40:13.350637 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:13.350656 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:40:13.386184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21757 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:40:15.587184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140387 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:40:15.872929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos 2013-04-03 10:40:15.872949 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 2013-04-03 10:40:15.872966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:16.937406 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:16.937415 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c5340 from 10.252.24.88:6789/0 2013-04-03 10:40:16.937470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 102 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:40:17.386317 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21758 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:40:18.350773 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29051 v29051: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 38371B/s wr, 0op/s 2013-04-03 10:40:18.350814 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:40:18.350837 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:40:18.350879 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:40:18.350910 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos 2013-04-03 10:40:18.350929 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 2013-04-03 10:40:18.350938 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394052 log 2013-04-03 10:40:18.350955 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:18.350974 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:40:18.872973 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos 2013-04-03 10:40:18.872993 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 2013-04-03 10:40:18.873010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:19.587332 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140388 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:40:20.933486 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:20.933496 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c58c0 from 10.252.24.88:6789/0 2013-04-03 10:40:20.933549 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 103 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:40:21.386467 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21759 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:40:21.594493 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:21.594502 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:40:21.594537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 preprocess_query pg_stats(1 pgs tid 34 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:40:21.594561 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 prepare_update pg_stats(1 pgs tid 34 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:40:21.594567 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 prepare_pg_stats pg_stats(1 pgs tid 34 v 605) v1 from osd.1 2013-04-03 10:40:21.594573 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:40:21.644758 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29051 encode_pending v 29052 2013-04-03 10:40:21.767558 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29051 update_from_paxos applying incremental 29052 2013-04-03 10:40:21.767577 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 v29052: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 306B/s wr, 0op/s 2013-04-03 10:40:21.767596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 send_pg_creates to 0 pgs 2013-04-03 10:40:21.767598 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 update_logger 2013-04-03 10:40:21.767608 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 create_pending v 29053 2013-04-03 10:40:21.767616 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29052 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:40:21.767634 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 check_osd_map already seen 605 >= 605 2013-04-03 10:40:21.767637 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 update_logger 2013-04-03 10:40:21.767647 7fcdffcdb700 0 log [INF] : pgmap v29052: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 306B/s wr, 0op/s 2013-04-03 10:40:21.767710 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos 2013-04-03 10:40:21.767729 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 2013-04-03 10:40:21.767748 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:21.767790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos 2013-04-03 10:40:21.767808 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 2013-04-03 10:40:21.767817 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:21.767828 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:40:21.767852 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:21.767864 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:40:21.767874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 logging 2013-04-03 10:40:21.767648 mon.0 10.252.24.25:6789/0 118 : [INF] pgmap v29052: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 306B/s wr, 0op/s 2013-04-03 10:40:21.780347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos 2013-04-03 10:40:21.780363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 2013-04-03 10:40:21.780377 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:21.780652 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:21.780660 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:40:21.780687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 preprocess_query pg_stats(1 pgs tid 30449 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:40:21.780707 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 prepare_update pg_stats(1 pgs tid 30449 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:40:21.780713 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 prepare_pg_stats pg_stats(1 pgs tid 30449 v 605) v1 from osd.3 2013-04-03 10:40:21.780718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:40:22.767291 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29052 encode_pending v 29053 2013-04-03 10:40:22.769947 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394052 encode_pending v394053 2013-04-03 10:40:22.795342 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29052 update_from_paxos applying incremental 29053 2013-04-03 10:40:22.795361 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 v29053: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 467B/s wr, 0op/s 2013-04-03 10:40:22.795380 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 send_pg_creates to 0 pgs 2013-04-03 10:40:22.795383 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 update_logger 2013-04-03 10:40:22.795392 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 create_pending v 29054 2013-04-03 10:40:22.795400 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29053 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:40:22.795418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 check_osd_map already seen 605 >= 605 2013-04-03 10:40:22.795421 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 update_logger 2013-04-03 10:40:22.795430 7fcdffcdb700 0 log [INF] : pgmap v29053: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 467B/s wr, 0op/s 2013-04-03 10:40:22.834209 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos 2013-04-03 10:40:22.834233 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394052 2013-04-03 10:40:22.834249 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos latest full 393798 2013-04-03 10:40:22.834266 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394053 update_from_paxos applying incremental log 394053 2013-04-03 10:40:21.767648 mon.0 10.252.24.25:6789/0 118 : [INF] pgmap v29052: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 306B/s wr, 0op/s 2013-04-03 10:40:22.834309 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 check_subs 2013-04-03 10:40:22.834340 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 create_pending v 394054 2013-04-03 10:40:22.834362 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394053 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:40:22.834394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos 2013-04-03 10:40:22.834408 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 2013-04-03 10:40:22.834418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:22.834430 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:40:22.834447 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 nothing new 2013-04-03 10:40:22.834494 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos 2013-04-03 10:40:22.834509 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 2013-04-03 10:40:22.834527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:22.845178 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos 2013-04-03 10:40:22.845195 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 2013-04-03 10:40:22.845211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:23.351120 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29053 check_down_pgs 2013-04-03 10:40:23.351150 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29053 v29053: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 467B/s wr, 0op/s 2013-04-03 10:40:23.351178 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:40:23.351203 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:40:23.351246 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:40:23.351277 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos 2013-04-03 10:40:23.351296 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 2013-04-03 10:40:23.351305 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394053 log 2013-04-03 10:40:23.351322 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:23.351341 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:40:23.587413 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140389 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:40:24.933658 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:24.933667 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2995dc0 from 10.252.24.88:6789/0 2013-04-03 10:40:24.933719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 104 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:40:25.386584 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21760 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:40:25.835190 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos 2013-04-03 10:40:25.835210 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 2013-04-03 10:40:25.835227 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:27.587614 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140390 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:40:28.236589 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:28.236598 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:40:28.236648 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 preprocess_query pg_stats(1 pgs tid 14 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:40:28.236672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 prepare_update pg_stats(1 pgs tid 14 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:40:28.236678 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 prepare_pg_stats pg_stats(1 pgs tid 14 v 605) v1 from osd.2 2013-04-03 10:40:28.236684 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:40:28.286888 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29053 encode_pending v 29054 2013-04-03 10:40:28.315978 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29053 update_from_paxos applying incremental 29054 2013-04-03 10:40:28.315997 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29054 v29054: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2620B/s wr, 0op/s 2013-04-03 10:40:28.316017 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29054 send_pg_creates to 0 pgs 2013-04-03 10:40:28.316023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29054 update_logger 2013-04-03 10:40:28.316033 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29054 create_pending v 29055 2013-04-03 10:40:28.316042 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29054 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:40:28.316059 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29054 check_osd_map already seen 605 >= 605 2013-04-03 10:40:28.316062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29054 update_logger 2013-04-03 10:40:28.316072 7fcdffcdb700 0 log [INF] : pgmap v29054: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2620B/s wr, 0op/s 2013-04-03 10:40:28.316138 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos 2013-04-03 10:40:28.316157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 2013-04-03 10:40:28.316175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:28.316219 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos 2013-04-03 10:40:28.316233 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 2013-04-03 10:40:28.316242 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:28.316256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:40:28.316286 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:28.316298 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:40:28.316308 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 logging 2013-04-03 10:40:22.795431 mon.0 10.252.24.25:6789/0 119 : [INF] pgmap v29053: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 467B/s wr, 0op/s 2013-04-03 10:40:28.316329 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 logging 2013-04-03 10:40:28.316073 mon.0 10.252.24.25:6789/0 120 : [INF] pgmap v29054: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2620B/s wr, 0op/s 2013-04-03 10:40:28.340659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos 2013-04-03 10:40:28.340675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 2013-04-03 10:40:28.340690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:28.351456 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29054 check_down_pgs 2013-04-03 10:40:28.351491 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29054 v29054: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2620B/s wr, 0op/s 2013-04-03 10:40:28.351514 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:40:28.351536 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:40:28.351572 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:40:28.351602 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos 2013-04-03 10:40:28.351619 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 2013-04-03 10:40:28.351628 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394053 log 2013-04-03 10:40:28.351643 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:40:28.351658 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:40:28.541256 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:40:28.541261 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:40:28.541268 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 2013-04-03 10:40:28.933736 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:28.933747 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2995840 from 10.252.24.88:6789/0 2013-04-03 10:40:28.933800 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 105 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:40:29.315793 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394053 encode_pending v394054 2013-04-03 10:40:29.386636 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21761 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:40:29.388956 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos 2013-04-03 10:40:29.388985 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394053 2013-04-03 10:40:29.389002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos latest full 393798 2013-04-03 10:40:29.389021 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos applying incremental log 394054 2013-04-03 10:40:22.795431 mon.0 10.252.24.25:6789/0 119 : [INF] pgmap v29053: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 467B/s wr, 0op/s 2013-04-03 10:40:29.389049 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos applying incremental log 394054 2013-04-03 10:40:28.316073 mon.0 10.252.24.25:6789/0 120 : [INF] pgmap v29054: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2620B/s wr, 0op/s 2013-04-03 10:40:29.389101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 check_subs 2013-04-03 10:40:29.389135 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 create_pending v 394055 2013-04-03 10:40:29.389158 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:40:29.389219 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos 2013-04-03 10:40:29.389235 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394054 2013-04-03 10:40:29.389254 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:29.401925 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos 2013-04-03 10:40:29.401941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394054 2013-04-03 10:40:29.401958 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:30.681227 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2c80 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33078/0) 2013-04-03 10:40:31.587733 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140391 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:40:32.389879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos 2013-04-03 10:40:32.389900 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394054 2013-04-03 10:40:32.389919 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:32.933975 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:32.933986 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a84600 from 10.252.24.88:6789/0 2013-04-03 10:40:32.934032 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 106 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:40:33.237182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:33.237190 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:40:33.237225 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29054 preprocess_query pg_stats(1 pgs tid 15 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:40:33.237249 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29054 prepare_update pg_stats(1 pgs tid 15 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:40:33.237255 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29054 prepare_pg_stats pg_stats(1 pgs tid 15 v 605) v1 from osd.2 2013-04-03 10:40:33.237261 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29054 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:40:33.287469 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29054 encode_pending v 29055 2013-04-03 10:40:33.315233 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29054 update_from_paxos applying incremental 29055 2013-04-03 10:40:33.315253 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 v29055: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1946B/s wr, 0op/s 2013-04-03 10:40:33.315273 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 send_pg_creates to 0 pgs 2013-04-03 10:40:33.315277 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 update_logger 2013-04-03 10:40:33.315287 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 create_pending v 29056 2013-04-03 10:40:33.315299 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:40:33.315315 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 check_osd_map already seen 605 >= 605 2013-04-03 10:40:33.315319 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 update_logger 2013-04-03 10:40:33.315330 7fcdffcdb700 0 log [INF] : pgmap v29055: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1946B/s wr, 0op/s 2013-04-03 10:40:33.315390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos 2013-04-03 10:40:33.315410 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394054 2013-04-03 10:40:33.315429 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:33.315474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos 2013-04-03 10:40:33.315489 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394054 2013-04-03 10:40:33.315498 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:33.315510 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:40:33.315534 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:33.315547 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:40:33.315557 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 logging 2013-04-03 10:40:33.315331 mon.0 10.252.24.25:6789/0 121 : [INF] pgmap v29055: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1946B/s wr, 0op/s 2013-04-03 10:40:33.339896 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos 2013-04-03 10:40:33.339913 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394054 2013-04-03 10:40:33.339928 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:33.351771 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 check_down_pgs 2013-04-03 10:40:33.351808 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 v29055: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1946B/s wr, 0op/s 2013-04-03 10:40:33.351832 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:40:33.351854 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:40:33.351891 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:40:33.351922 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos 2013-04-03 10:40:33.351940 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394054 2013-04-03 10:40:33.351949 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 log 2013-04-03 10:40:33.351965 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:33.351983 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:40:33.386738 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21762 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:40:34.314989 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 encode_pending v394055 2013-04-03 10:40:34.339496 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos 2013-04-03 10:40:34.339522 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos version 394055 summary v 394054 2013-04-03 10:40:34.339539 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos latest full 393798 2013-04-03 10:40:34.339559 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos applying incremental log 394055 2013-04-03 10:40:33.315331 mon.0 10.252.24.25:6789/0 121 : [INF] pgmap v29055: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1946B/s wr, 0op/s 2013-04-03 10:40:34.339606 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 check_subs 2013-04-03 10:40:34.339639 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 create_pending v 394056 2013-04-03 10:40:34.339662 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:40:34.339728 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos 2013-04-03 10:40:34.339744 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos version 394055 summary v 394055 2013-04-03 10:40:34.339763 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:34.351781 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos 2013-04-03 10:40:34.351798 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos version 394055 summary v 394055 2013-04-03 10:40:34.351815 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:35.587842 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140392 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:40:36.596162 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:36.596173 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:40:36.596223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 preprocess_query pg_stats(1 pgs tid 35 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:40:36.596248 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 prepare_update pg_stats(1 pgs tid 35 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:40:36.596255 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 prepare_pg_stats pg_stats(1 pgs tid 35 v 605) v1 from osd.1 2013-04-03 10:40:36.596261 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:40:36.646446 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 encode_pending v 29056 2013-04-03 10:40:36.700067 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 update_from_paxos applying incremental 29056 2013-04-03 10:40:36.700084 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 v29056: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 979B/s wr, 0op/s 2013-04-03 10:40:36.700104 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 send_pg_creates to 0 pgs 2013-04-03 10:40:36.700109 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 update_logger 2013-04-03 10:40:36.700119 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 create_pending v 29057 2013-04-03 10:40:36.700129 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:40:36.700145 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 check_osd_map already seen 605 >= 605 2013-04-03 10:40:36.700149 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 update_logger 2013-04-03 10:40:36.700166 7fcdffcdb700 0 log [INF] : pgmap v29056: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 979B/s wr, 0op/s 2013-04-03 10:40:36.700228 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos 2013-04-03 10:40:36.700249 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos version 394055 summary v 394055 2013-04-03 10:40:36.700267 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:36.700314 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos 2013-04-03 10:40:36.700329 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos version 394055 summary v 394055 2013-04-03 10:40:36.700339 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:36.700351 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:40:36.700375 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:36.700387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:40:36.700398 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 logging 2013-04-03 10:40:36.700169 mon.0 10.252.24.25:6789/0 122 : [INF] pgmap v29056: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 979B/s wr, 0op/s 2013-04-03 10:40:36.723638 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos 2013-04-03 10:40:36.723655 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos version 394055 summary v 394055 2013-04-03 10:40:36.723674 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:36.934013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:36.934024 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2995b00 from 10.252.24.88:6789/0 2013-04-03 10:40:36.934075 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 107 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:40:37.386871 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21763 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:40:37.699829 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 encode_pending v394056 2013-04-03 10:40:37.731468 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos 2013-04-03 10:40:37.731496 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394055 2013-04-03 10:40:37.731511 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos latest full 393798 2013-04-03 10:40:37.731530 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos applying incremental log 394056 2013-04-03 10:40:36.700169 mon.0 10.252.24.25:6789/0 122 : [INF] pgmap v29056: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 979B/s wr, 0op/s 2013-04-03 10:40:37.731576 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 check_subs 2013-04-03 10:40:37.731608 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 create_pending v 394057 2013-04-03 10:40:37.731632 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:40:37.731697 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos 2013-04-03 10:40:37.731713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 2013-04-03 10:40:37.731730 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:37.744331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos 2013-04-03 10:40:37.744348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 2013-04-03 10:40:37.744364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:38.352122 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 check_down_pgs 2013-04-03 10:40:38.352157 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 v29056: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 979B/s wr, 0op/s 2013-04-03 10:40:38.352187 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:40:38.352212 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:40:38.352249 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:40:38.352282 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos 2013-04-03 10:40:38.352301 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 2013-04-03 10:40:38.352311 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 log 2013-04-03 10:40:38.352328 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:38.352348 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:40:39.587987 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140393 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:40:40.697215 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2a00 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33079/0) 2013-04-03 10:40:40.732296 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos 2013-04-03 10:40:40.732316 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 2013-04-03 10:40:40.732334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:40.934134 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:40.934146 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2996600 from 10.252.24.88:6789/0 2013-04-03 10:40:40.934199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 108 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:40:41.386977 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21764 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:40:43.352491 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 v29056: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 979B/s wr, 0op/s 2013-04-03 10:40:43.352540 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:40:43.352564 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:40:43.352610 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:40:43.352642 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos 2013-04-03 10:40:43.352662 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 2013-04-03 10:40:43.352671 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 log 2013-04-03 10:40:43.352689 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:43.352709 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:40:43.588060 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140394 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:40:43.732349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos 2013-04-03 10:40:43.732367 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 2013-04-03 10:40:43.732384 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:44.934270 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:44.934281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29968c0 from 10.252.24.88:6789/0 2013-04-03 10:40:44.934334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 109 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:40:45.387077 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21765 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:40:46.597288 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:46.597299 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:40:46.597349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 preprocess_query pg_stats(1 pgs tid 36 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:40:46.597374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 prepare_update pg_stats(1 pgs tid 36 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:40:46.597381 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 prepare_pg_stats pg_stats(1 pgs tid 36 v 605) v1 from osd.1 2013-04-03 10:40:46.597387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:40:46.647585 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 encode_pending v 29057 2013-04-03 10:40:46.721659 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 update_from_paxos applying incremental 29057 2013-04-03 10:40:46.721678 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 v29057: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 613B/s wr, 0op/s 2013-04-03 10:40:46.721698 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 send_pg_creates to 0 pgs 2013-04-03 10:40:46.721702 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 update_logger 2013-04-03 10:40:46.721713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 create_pending v 29058 2013-04-03 10:40:46.721722 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:40:46.721740 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 check_osd_map already seen 605 >= 605 2013-04-03 10:40:46.721744 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 update_logger 2013-04-03 10:40:46.721755 7fcdffcdb700 0 log [INF] : pgmap v29057: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 613B/s wr, 0op/s 2013-04-03 10:40:46.721823 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos 2013-04-03 10:40:46.721844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 2013-04-03 10:40:46.721863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:46.721907 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos 2013-04-03 10:40:46.721923 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 2013-04-03 10:40:46.721932 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:46.721944 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:40:46.721968 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:46.721980 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:40:46.721991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 logging 2013-04-03 10:40:46.721756 mon.0 10.252.24.25:6789/0 123 : [INF] pgmap v29057: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 613B/s wr, 0op/s 2013-04-03 10:40:46.747310 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos 2013-04-03 10:40:46.747327 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 2013-04-03 10:40:46.747342 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:46.762797 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:46.762805 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:40:46.762834 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 preprocess_query pg_stats(1 pgs tid 30450 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:40:46.762855 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 prepare_update pg_stats(1 pgs tid 30450 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:40:46.762861 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 prepare_pg_stats pg_stats(1 pgs tid 30450 v 605) v1 from osd.3 2013-04-03 10:40:46.762867 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:40:47.588209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140395 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:40:47.721381 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 encode_pending v394057 2013-04-03 10:40:47.722810 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 encode_pending v 29058 2013-04-03 10:40:47.760748 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos 2013-04-03 10:40:47.760775 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos version 394057 summary v 394056 2013-04-03 10:40:47.760794 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos latest full 393798 2013-04-03 10:40:47.760813 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos applying incremental log 394057 2013-04-03 10:40:46.721756 mon.0 10.252.24.25:6789/0 123 : [INF] pgmap v29057: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 613B/s wr, 0op/s 2013-04-03 10:40:47.760859 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 check_subs 2013-04-03 10:40:47.760891 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 create_pending v 394058 2013-04-03 10:40:47.760914 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:40:47.794416 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 update_from_paxos applying incremental 29058 2013-04-03 10:40:47.794436 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 v29058: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 462B/s wr, 0op/s 2013-04-03 10:40:47.794456 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 send_pg_creates to 0 pgs 2013-04-03 10:40:47.794459 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 update_logger 2013-04-03 10:40:47.794474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 create_pending v 29059 2013-04-03 10:40:47.794486 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:40:47.794503 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 check_osd_map already seen 605 >= 605 2013-04-03 10:40:47.794506 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 update_logger 2013-04-03 10:40:47.794516 7fcdffcdb700 0 log [INF] : pgmap v29058: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 462B/s wr, 0op/s 2013-04-03 10:40:47.794572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos 2013-04-03 10:40:47.794590 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos version 394057 summary v 394057 2013-04-03 10:40:47.794608 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:47.794655 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos 2013-04-03 10:40:47.794671 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos version 394057 summary v 394057 2013-04-03 10:40:47.794680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:47.794692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:40:47.794711 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:47.794728 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:40:47.794738 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 logging 2013-04-03 10:40:47.794517 mon.0 10.252.24.25:6789/0 124 : [INF] pgmap v29058: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 462B/s wr, 0op/s 2013-04-03 10:40:47.854660 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos 2013-04-03 10:40:47.854677 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos version 394057 summary v 394057 2013-04-03 10:40:47.854692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:48.352852 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 check_down_pgs 2013-04-03 10:40:48.352889 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 v29058: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 462B/s wr, 0op/s 2013-04-03 10:40:48.352922 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:40:48.352947 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:40:48.352984 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:40:48.353016 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos 2013-04-03 10:40:48.353036 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos version 394057 summary v 394057 2013-04-03 10:40:48.353046 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 log 2013-04-03 10:40:48.353063 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:48.353083 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:40:48.794211 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 encode_pending v394058 2013-04-03 10:40:48.821698 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos 2013-04-03 10:40:48.821726 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394057 2013-04-03 10:40:48.821742 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos latest full 393798 2013-04-03 10:40:48.821762 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos applying incremental log 394058 2013-04-03 10:40:47.794517 mon.0 10.252.24.25:6789/0 124 : [INF] pgmap v29058: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 462B/s wr, 0op/s 2013-04-03 10:40:48.821807 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 check_subs 2013-04-03 10:40:48.821847 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 create_pending v 394059 2013-04-03 10:40:48.821871 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:40:48.821931 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos 2013-04-03 10:40:48.821947 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394058 2013-04-03 10:40:48.821968 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:48.835018 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos 2013-04-03 10:40:48.835035 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394058 2013-04-03 10:40:48.835052 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:48.934355 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:48.934364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83000 from 10.252.24.88:6789/0 2013-04-03 10:40:48.934397 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 110 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:40:49.387193 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21766 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:40:50.713219 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33080/0) 2013-04-03 10:40:51.588343 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140396 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:40:51.822542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos 2013-04-03 10:40:51.822563 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394058 2013-04-03 10:40:51.822580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:52.934547 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:52.934557 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83dc0 from 10.252.24.88:6789/0 2013-04-03 10:40:52.934611 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 111 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:40:53.239750 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:53.239761 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:40:53.239810 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 preprocess_query pg_stats(1 pgs tid 16 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:40:53.239835 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 prepare_update pg_stats(1 pgs tid 16 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:40:53.239841 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 prepare_pg_stats pg_stats(1 pgs tid 16 v 605) v1 from osd.2 2013-04-03 10:40:53.239851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:40:53.290070 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 encode_pending v 29059 2013-04-03 10:40:53.336107 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 update_from_paxos applying incremental 29059 2013-04-03 10:40:53.336128 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 v29059: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3237B/s wr, 0op/s 2013-04-03 10:40:53.336149 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 send_pg_creates to 0 pgs 2013-04-03 10:40:53.336154 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 update_logger 2013-04-03 10:40:53.336164 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 create_pending v 29060 2013-04-03 10:40:53.336174 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:40:53.336192 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 check_osd_map already seen 605 >= 605 2013-04-03 10:40:53.336194 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 update_logger 2013-04-03 10:40:53.336205 7fcdffcdb700 0 log [INF] : pgmap v29059: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3237B/s wr, 0op/s 2013-04-03 10:40:53.336265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos 2013-04-03 10:40:53.336288 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394058 2013-04-03 10:40:53.336308 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:53.336352 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos 2013-04-03 10:40:53.336367 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394058 2013-04-03 10:40:53.336376 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:53.336389 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:40:53.336413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:40:53.336426 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:40:53.336436 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 logging 2013-04-03 10:40:53.336206 mon.0 10.252.24.25:6789/0 125 : [INF] pgmap v29059: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3237B/s wr, 0op/s 2013-04-03 10:40:53.346254 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos 2013-04-03 10:40:53.346271 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394058 2013-04-03 10:40:53.346286 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:53.353182 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 check_down_pgs 2013-04-03 10:40:53.353211 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 v29059: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3237B/s wr, 0op/s 2013-04-03 10:40:53.353242 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:40:53.353264 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:40:53.353300 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:40:53.353339 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos 2013-04-03 10:40:53.353357 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394058 2013-04-03 10:40:53.353366 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 log 2013-04-03 10:40:53.353383 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:53.353399 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:40:53.387229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21767 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:40:54.317431 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 encode_pending v394059 2013-04-03 10:40:54.346677 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos 2013-04-03 10:40:54.346705 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos version 394059 summary v 394058 2013-04-03 10:40:54.346719 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos latest full 393798 2013-04-03 10:40:54.346738 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos applying incremental log 394059 2013-04-03 10:40:53.336206 mon.0 10.252.24.25:6789/0 125 : [INF] pgmap v29059: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3237B/s wr, 0op/s 2013-04-03 10:40:54.346785 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 check_subs 2013-04-03 10:40:54.346817 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 create_pending v 394060 2013-04-03 10:40:54.346841 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:40:54.346906 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos 2013-04-03 10:40:54.346923 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos version 394059 summary v 394059 2013-04-03 10:40:54.346941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:54.358717 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos 2013-04-03 10:40:54.358734 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos version 394059 summary v 394059 2013-04-03 10:40:54.358751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:55.588472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140397 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:40:56.934676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:40:56.934687 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83580 from 10.252.24.88:6789/0 2013-04-03 10:40:56.934741 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 112 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:40:57.347717 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos 2013-04-03 10:40:57.347738 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos version 394059 summary v 394059 2013-04-03 10:40:57.347756 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:57.387330 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21768 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:40:58.353542 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 v29059: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3237B/s wr, 0op/s 2013-04-03 10:40:58.353590 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:40:58.353614 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:40:58.353660 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:40:58.353691 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos 2013-04-03 10:40:58.353996 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos version 394059 summary v 394059 2013-04-03 10:40:58.354012 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 log 2013-04-03 10:40:58.354030 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:40:58.354050 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:40:58.541332 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) 2013-04-03 10:40:58.541361 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader 2013-04-03 10:40:59.588575 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140398 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:00.347681 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos 2013-04-03 10:41:00.347701 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos version 394059 summary v 394059 2013-04-03 10:41:00.347718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:00.934713 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:00.934721 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83b00 from 10.252.24.88:6789/0 2013-04-03 10:41:00.934756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 113 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:41:01.387515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21769 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:41:01.598985 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:01.598994 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:41:01.599027 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29059 preprocess_query pg_stats(1 pgs tid 37 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:41:01.599051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29059 prepare_update pg_stats(1 pgs tid 37 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:41:01.599057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29059 prepare_pg_stats pg_stats(1 pgs tid 37 v 605) v1 from osd.1 2013-04-03 10:41:01.599062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29059 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:41:01.649248 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29059 encode_pending v 29060 2013-04-03 10:41:01.817252 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29059 update_from_paxos applying incremental 29060 2013-04-03 10:41:01.817268 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 v29060: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1764B/s wr, 0op/s 2013-04-03 10:41:01.817287 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 send_pg_creates to 0 pgs 2013-04-03 10:41:01.817290 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 update_logger 2013-04-03 10:41:01.817299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 create_pending v 29061 2013-04-03 10:41:01.817308 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29060 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:41:01.817324 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 check_osd_map already seen 605 >= 605 2013-04-03 10:41:01.817328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 update_logger 2013-04-03 10:41:01.817338 7fcdffcdb700 0 log [INF] : pgmap v29060: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1764B/s wr, 0op/s 2013-04-03 10:41:01.817396 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos 2013-04-03 10:41:01.817416 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos version 394059 summary v 394059 2013-04-03 10:41:01.817434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:01.817479 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos 2013-04-03 10:41:01.817494 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos version 394059 summary v 394059 2013-04-03 10:41:01.817503 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:01.817514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:41:01.817538 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:01.817551 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:41:01.817562 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 logging 2013-04-03 10:41:01.817339 mon.0 10.252.24.25:6789/0 126 : [INF] pgmap v29060: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1764B/s wr, 0op/s 2013-04-03 10:41:01.842505 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos 2013-04-03 10:41:01.842523 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos version 394059 summary v 394059 2013-04-03 10:41:01.842538 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:02.817031 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394059 encode_pending v394060 2013-04-03 10:41:02.866303 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos 2013-04-03 10:41:02.866329 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos version 394060 summary v 394059 2013-04-03 10:41:02.866344 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos latest full 393798 2013-04-03 10:41:02.866362 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394060 update_from_paxos applying incremental log 394060 2013-04-03 10:41:01.817339 mon.0 10.252.24.25:6789/0 126 : [INF] pgmap v29060: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1764B/s wr, 0op/s 2013-04-03 10:41:02.866411 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 check_subs 2013-04-03 10:41:02.866442 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 create_pending v 394061 2013-04-03 10:41:02.866464 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394060 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:41:02.866529 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos 2013-04-03 10:41:02.866545 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos version 394060 summary v 394060 2013-04-03 10:41:02.866563 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:02.891606 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos 2013-04-03 10:41:02.891623 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos version 394060 summary v 394060 2013-04-03 10:41:02.891638 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:03.241280 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:03.241291 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:41:03.241341 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 preprocess_query pg_stats(1 pgs tid 17 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:41:03.241365 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 prepare_update pg_stats(1 pgs tid 17 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:41:03.241372 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 prepare_pg_stats pg_stats(1 pgs tid 17 v 605) v1 from osd.2 2013-04-03 10:41:03.241379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:41:03.354195 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29060 check_down_pgs 2013-04-03 10:41:03.354224 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29060 v29060: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1764B/s wr, 0op/s 2013-04-03 10:41:03.354254 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:41:03.354279 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:41:03.354315 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:41:03.354346 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos 2013-04-03 10:41:03.354364 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos version 394060 summary v 394060 2013-04-03 10:41:03.354373 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394060 log 2013-04-03 10:41:03.354390 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:03.354409 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:41:03.588674 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140399 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:03.865677 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29060 encode_pending v 29061 2013-04-03 10:41:03.895436 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29060 update_from_paxos applying incremental 29061 2013-04-03 10:41:03.895455 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 v29061: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s 2013-04-03 10:41:03.895473 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 send_pg_creates to 0 pgs 2013-04-03 10:41:03.895475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 update_logger 2013-04-03 10:41:03.895485 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 create_pending v 29062 2013-04-03 10:41:03.895496 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29061 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:41:03.895515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 check_osd_map already seen 605 >= 605 2013-04-03 10:41:03.895519 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 update_logger 2013-04-03 10:41:03.895529 7fcdffcdb700 0 log [INF] : pgmap v29061: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s 2013-04-03 10:41:03.895595 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos 2013-04-03 10:41:03.895613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos version 394060 summary v 394060 2013-04-03 10:41:03.895630 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:03.895672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos 2013-04-03 10:41:03.895687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos version 394060 summary v 394060 2013-04-03 10:41:03.895696 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:03.895707 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:41:03.895731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:03.895744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:41:03.895754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 logging 2013-04-03 10:41:03.895530 mon.0 10.252.24.25:6789/0 127 : [INF] pgmap v29061: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s 2013-04-03 10:41:03.907247 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos 2013-04-03 10:41:03.907263 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos version 394060 summary v 394060 2013-04-03 10:41:03.907280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:04.895222 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394060 encode_pending v394061 2013-04-03 10:41:04.932718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos 2013-04-03 10:41:04.932744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394060 2013-04-03 10:41:04.932760 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos latest full 393798 2013-04-03 10:41:04.932779 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394061 update_from_paxos applying incremental log 394061 2013-04-03 10:41:03.895530 mon.0 10.252.24.25:6789/0 127 : [INF] pgmap v29061: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s 2013-04-03 10:41:04.932823 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 check_subs 2013-04-03 10:41:04.932854 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 create_pending v 394062 2013-04-03 10:41:04.932876 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394061 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:41:04.932940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos 2013-04-03 10:41:04.932955 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 2013-04-03 10:41:04.932973 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:04.958416 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos 2013-04-03 10:41:04.958432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 2013-04-03 10:41:04.958450 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:04.958715 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:04.958724 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e3dc0 from 10.252.24.88:6789/0 2013-04-03 10:41:04.958751 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 114 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:41:05.387666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21770 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:41:07.588829 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140400 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:07.933553 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos 2013-04-03 10:41:07.933573 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 2013-04-03 10:41:07.933590 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:08.354557 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29061 check_down_pgs 2013-04-03 10:41:08.354586 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29061 v29061: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s 2013-04-03 10:41:08.354614 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:41:08.354638 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:41:08.354675 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:41:08.354706 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos 2013-04-03 10:41:08.354724 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 2013-04-03 10:41:08.354734 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394061 log 2013-04-03 10:41:08.354751 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:08.354768 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:41:08.934998 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:08.935007 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4600 from 10.252.24.88:6789/0 2013-04-03 10:41:08.935050 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 115 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:41:09.387784 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21771 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:41:10.933640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos 2013-04-03 10:41:10.933660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 2013-04-03 10:41:10.933677 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:11.588952 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140401 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:11.600051 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:11.600059 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:41:11.600092 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 preprocess_query pg_stats(1 pgs tid 38 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:41:11.600115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 prepare_update pg_stats(1 pgs tid 38 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:41:11.600121 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 prepare_pg_stats pg_stats(1 pgs tid 38 v 605) v1 from osd.1 2013-04-03 10:41:11.600127 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:41:11.650313 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29061 encode_pending v 29062 2013-04-03 10:41:11.683672 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29061 update_from_paxos applying incremental 29062 2013-04-03 10:41:11.683688 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 v29062: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s 2013-04-03 10:41:11.683706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 send_pg_creates to 0 pgs 2013-04-03 10:41:11.683710 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 update_logger 2013-04-03 10:41:11.683720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 create_pending v 29063 2013-04-03 10:41:11.683728 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29062 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:41:11.683749 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 check_osd_map already seen 605 >= 605 2013-04-03 10:41:11.683752 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 update_logger 2013-04-03 10:41:11.683762 7fcdffcdb700 0 log [INF] : pgmap v29062: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s 2013-04-03 10:41:11.683819 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos 2013-04-03 10:41:11.683839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 2013-04-03 10:41:11.683859 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:11.683901 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos 2013-04-03 10:41:11.683916 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 2013-04-03 10:41:11.683924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:11.683933 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:41:11.683956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:11.683968 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:41:11.683977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 logging 2013-04-03 10:41:11.683763 mon.0 10.252.24.25:6789/0 128 : [INF] pgmap v29062: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s 2013-04-03 10:41:11.695744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos 2013-04-03 10:41:11.695762 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 2013-04-03 10:41:11.695777 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:11.765552 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:11.765560 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:41:11.765594 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 preprocess_query pg_stats(1 pgs tid 30451 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:41:11.765615 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 prepare_update pg_stats(1 pgs tid 30451 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:41:11.765621 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 prepare_pg_stats pg_stats(1 pgs tid 30451 v 605) v1 from osd.3 2013-04-03 10:41:11.765627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:41:12.683475 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29062 encode_pending v 29063 2013-04-03 10:41:12.684924 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394061 encode_pending v394062 2013-04-03 10:41:12.837791 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29062 update_from_paxos applying incremental 29063 2013-04-03 10:41:12.837817 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 v29063: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s 2013-04-03 10:41:12.837838 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 send_pg_creates to 0 pgs 2013-04-03 10:41:12.837842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 update_logger 2013-04-03 10:41:12.837852 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 create_pending v 29064 2013-04-03 10:41:12.837861 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29063 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:41:12.837878 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 check_osd_map already seen 605 >= 605 2013-04-03 10:41:12.837881 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 update_logger 2013-04-03 10:41:12.837891 7fcdffcdb700 0 log [INF] : pgmap v29063: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s 2013-04-03 10:41:12.877282 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos 2013-04-03 10:41:12.877306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394061 2013-04-03 10:41:12.877320 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos latest full 393798 2013-04-03 10:41:12.877339 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394062 update_from_paxos applying incremental log 394062 2013-04-03 10:41:11.683763 mon.0 10.252.24.25:6789/0 128 : [INF] pgmap v29062: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s 2013-04-03 10:41:12.877384 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 check_subs 2013-04-03 10:41:12.877415 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 create_pending v 394063 2013-04-03 10:41:12.877437 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394062 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:41:12.877471 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos 2013-04-03 10:41:12.877486 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 2013-04-03 10:41:12.877495 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:12.877506 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:41:12.877524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 nothing new 2013-04-03 10:41:12.877573 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos 2013-04-03 10:41:12.877588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 2013-04-03 10:41:12.877606 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:12.889357 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos 2013-04-03 10:41:12.889374 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 2013-04-03 10:41:12.889389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:12.935066 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:12.935074 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e3b00 from 10.252.24.88:6789/0 2013-04-03 10:41:12.935105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 116 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:41:13.354912 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29063 check_down_pgs 2013-04-03 10:41:13.354947 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29063 v29063: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s 2013-04-03 10:41:13.355256 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:41:13.355280 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:41:13.355318 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:41:13.355349 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos 2013-04-03 10:41:13.355367 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 2013-04-03 10:41:13.355376 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394062 log 2013-04-03 10:41:13.355393 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:13.355412 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:41:13.387856 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21772 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:41:15.589056 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140402 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:15.878155 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos 2013-04-03 10:41:15.878175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 2013-04-03 10:41:15.878192 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:16.935228 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:16.935238 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2145b00 from 10.252.24.88:6789/0 2013-04-03 10:41:16.935291 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 117 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:41:17.387977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21773 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:41:17.417989 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 2013-04-03 10:41:17.417997 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:41:18.243232 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:18.243240 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:41:18.243277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 preprocess_query pg_stats(1 pgs tid 18 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:41:18.243299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 prepare_update pg_stats(1 pgs tid 18 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:41:18.243303 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 prepare_pg_stats pg_stats(1 pgs tid 18 v 605) v1 from osd.2 2013-04-03 10:41:18.243309 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:41:18.293500 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29063 encode_pending v 29064 2013-04-03 10:41:18.320029 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29063 update_from_paxos applying incremental 29064 2013-04-03 10:41:18.320049 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 v29064: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:41:18.320068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 send_pg_creates to 0 pgs 2013-04-03 10:41:18.320072 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 update_logger 2013-04-03 10:41:18.320082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 create_pending v 29065 2013-04-03 10:41:18.320091 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29064 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:41:18.320110 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 check_osd_map already seen 605 >= 605 2013-04-03 10:41:18.320115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 update_logger 2013-04-03 10:41:18.320126 7fcdffcdb700 0 log [INF] : pgmap v29064: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:41:18.320203 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos 2013-04-03 10:41:18.320222 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 2013-04-03 10:41:18.320241 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:18.320284 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos 2013-04-03 10:41:18.320299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 2013-04-03 10:41:18.320308 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:18.320319 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:41:18.320351 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:18.320363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:41:18.320374 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 logging 2013-04-03 10:41:12.837892 mon.0 10.252.24.25:6789/0 129 : [INF] pgmap v29063: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s 2013-04-03 10:41:18.320396 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 logging 2013-04-03 10:41:18.320134 mon.0 10.252.24.25:6789/0 130 : [INF] pgmap v29064: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:41:18.345722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos 2013-04-03 10:41:18.345740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 2013-04-03 10:41:18.345756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:18.355526 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29064 check_down_pgs 2013-04-03 10:41:18.355561 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29064 v29064: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:41:18.355584 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:41:18.355606 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:41:18.355643 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:41:18.355673 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos 2013-04-03 10:41:18.355690 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 2013-04-03 10:41:18.355699 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394062 log 2013-04-03 10:41:18.355715 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:18.355732 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:41:19.319848 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394062 encode_pending v394063 2013-04-03 10:41:19.346108 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos 2013-04-03 10:41:19.346135 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394062 2013-04-03 10:41:19.346149 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos latest full 393798 2013-04-03 10:41:19.346168 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394063 update_from_paxos applying incremental log 394063 2013-04-03 10:41:12.837892 mon.0 10.252.24.25:6789/0 129 : [INF] pgmap v29063: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s 2013-04-03 10:41:19.346196 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394063 update_from_paxos applying incremental log 394063 2013-04-03 10:41:18.320134 mon.0 10.252.24.25:6789/0 130 : [INF] pgmap v29064: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:41:19.346235 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 check_subs 2013-04-03 10:41:19.346266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 create_pending v 394064 2013-04-03 10:41:19.346288 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394063 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:41:19.346353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos 2013-04-03 10:41:19.346368 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 2013-04-03 10:41:19.346386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:19.359502 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos 2013-04-03 10:41:19.359519 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 2013-04-03 10:41:19.359535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:19.589170 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140403 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:20.935417 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:20.935426 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x21452c0 from 10.252.24.88:6789/0 2013-04-03 10:41:20.935479 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 118 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:41:21.409058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21774 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:41:22.346983 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos 2013-04-03 10:41:22.347005 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 2013-04-03 10:41:22.347022 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:23.356017 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29064 v29064: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:41:23.356058 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:41:23.356084 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:41:23.356128 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:41:23.356159 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos 2013-04-03 10:41:23.356177 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 2013-04-03 10:41:23.356186 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394063 log 2013-04-03 10:41:23.356203 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:23.356222 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:41:23.589280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140404 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:24.935663 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:24.935672 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2145dc0 from 10.252.24.88:6789/0 2013-04-03 10:41:24.935725 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 119 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:41:25.347175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos 2013-04-03 10:41:25.347196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 2013-04-03 10:41:25.347213 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:25.388185 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21775 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:41:25.482953 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 2013-04-03 10:41:25.482964 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:41:26.601722 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:26.601732 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:41:26.601782 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 preprocess_query pg_stats(1 pgs tid 39 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:41:26.601806 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 prepare_update pg_stats(1 pgs tid 39 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:41:26.601812 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 prepare_pg_stats pg_stats(1 pgs tid 39 v 605) v1 from osd.1 2013-04-03 10:41:26.601818 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:41:26.652002 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29064 encode_pending v 29065 2013-04-03 10:41:26.696200 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29064 update_from_paxos applying incremental 29065 2013-04-03 10:41:26.696216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29065 v29065: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s 2013-04-03 10:41:26.696233 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29065 send_pg_creates to 0 pgs 2013-04-03 10:41:26.696237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29065 update_logger 2013-04-03 10:41:26.696247 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29065 create_pending v 29066 2013-04-03 10:41:26.696256 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29065 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:41:26.696274 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29065 check_osd_map already seen 605 >= 605 2013-04-03 10:41:26.696277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29065 update_logger 2013-04-03 10:41:26.696288 7fcdffcdb700 0 log [INF] : pgmap v29065: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s 2013-04-03 10:41:26.696348 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos 2013-04-03 10:41:26.696366 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 2013-04-03 10:41:26.696385 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:26.696427 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos 2013-04-03 10:41:26.696442 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 2013-04-03 10:41:26.696451 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:26.696462 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:41:26.696486 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:26.696498 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:41:26.696508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 logging 2013-04-03 10:41:26.696289 mon.0 10.252.24.25:6789/0 131 : [INF] pgmap v29065: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s 2013-04-03 10:41:26.721903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos 2013-04-03 10:41:26.721920 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 2013-04-03 10:41:26.721935 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:27.589429 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140405 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:27.695985 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394063 encode_pending v394064 2013-04-03 10:41:27.880445 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos 2013-04-03 10:41:27.880477 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos version 394064 summary v 394063 2013-04-03 10:41:27.880491 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos latest full 393798 2013-04-03 10:41:27.880509 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394064 update_from_paxos applying incremental log 394064 2013-04-03 10:41:26.696289 mon.0 10.252.24.25:6789/0 131 : [INF] pgmap v29065: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s 2013-04-03 10:41:27.880555 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 check_subs 2013-04-03 10:41:27.880587 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 create_pending v 394065 2013-04-03 10:41:27.880609 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394064 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:41:27.880673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos 2013-04-03 10:41:27.880689 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos version 394064 summary v 394064 2013-04-03 10:41:27.880711 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:27.893909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos 2013-04-03 10:41:27.893928 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos version 394064 summary v 394064 2013-04-03 10:41:27.893945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:28.356369 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29065 check_down_pgs 2013-04-03 10:41:28.356398 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29065 v29065: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s 2013-04-03 10:41:28.356426 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:41:28.356455 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:41:28.356492 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:41:28.356522 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos 2013-04-03 10:41:28.356541 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos version 394064 summary v 394064 2013-04-03 10:41:28.356550 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394064 log 2013-04-03 10:41:28.356567 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:28.356585 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:41:28.597413 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:41:28.597419 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:41:28.597428 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 2013-04-03 10:41:28.935609 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:28.935618 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x1bae600 from 10.252.24.88:6789/0 2013-04-03 10:41:28.935671 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 120 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:41:29.388263 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21776 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:41:30.777240 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2500 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33085/0) 2013-04-03 10:41:30.881357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos 2013-04-03 10:41:30.881378 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos version 394064 summary v 394064 2013-04-03 10:41:30.881396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:31.589584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140406 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:32.935795 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:32.935808 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x1bae340 from 10.252.24.88:6789/0 2013-04-03 10:41:32.935860 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 121 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:41:33.245200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:33.245208 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:41:33.245241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29065 preprocess_query pg_stats(1 pgs tid 19 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:41:33.245265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29065 prepare_update pg_stats(1 pgs tid 19 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:41:33.245273 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29065 prepare_pg_stats pg_stats(1 pgs tid 19 v 605) v1 from osd.2 2013-04-03 10:41:33.245279 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29065 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:41:33.295450 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29065 encode_pending v 29066 2013-04-03 10:41:33.326455 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29065 update_from_paxos applying incremental 29066 2013-04-03 10:41:33.326476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 v29066: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:41:33.326496 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 send_pg_creates to 0 pgs 2013-04-03 10:41:33.326500 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 update_logger 2013-04-03 10:41:33.326511 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 create_pending v 29067 2013-04-03 10:41:33.326520 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:41:33.326538 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 check_osd_map already seen 605 >= 605 2013-04-03 10:41:33.326541 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 update_logger 2013-04-03 10:41:33.326553 7fcdffcdb700 0 log [INF] : pgmap v29066: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:41:33.326626 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos 2013-04-03 10:41:33.326646 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos version 394064 summary v 394064 2013-04-03 10:41:33.326665 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:33.326709 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos 2013-04-03 10:41:33.326724 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos version 394064 summary v 394064 2013-04-03 10:41:33.326733 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:33.326746 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:41:33.326771 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:33.326784 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:41:33.326795 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 logging 2013-04-03 10:41:33.326561 mon.0 10.252.24.25:6789/0 132 : [INF] pgmap v29066: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:41:33.351205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos 2013-04-03 10:41:33.351222 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos version 394064 summary v 394064 2013-04-03 10:41:33.351238 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:33.356688 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 check_down_pgs 2013-04-03 10:41:33.356718 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 v29066: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:41:33.356748 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:41:33.356770 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:41:33.356807 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:41:33.356837 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos 2013-04-03 10:41:33.356854 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos version 394064 summary v 394064 2013-04-03 10:41:33.356865 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 log 2013-04-03 10:41:33.356881 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:33.356897 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:41:33.388354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21777 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:41:34.325297 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 encode_pending v394065 2013-04-03 10:41:34.359286 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos 2013-04-03 10:41:34.359314 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos version 394065 summary v 394064 2013-04-03 10:41:34.359329 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos latest full 393798 2013-04-03 10:41:34.359348 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos applying incremental log 394065 2013-04-03 10:41:33.326561 mon.0 10.252.24.25:6789/0 132 : [INF] pgmap v29066: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:41:34.359392 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 check_subs 2013-04-03 10:41:34.359424 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 create_pending v 394066 2013-04-03 10:41:34.359447 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:41:34.359512 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos 2013-04-03 10:41:34.359529 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos version 394065 summary v 394065 2013-04-03 10:41:34.359548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:34.391875 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos 2013-04-03 10:41:34.391893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos version 394065 summary v 394065 2013-04-03 10:41:34.391910 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:35.486105 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:35.486117 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:41:35.486171 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 preprocess_query pg_stats(0 pgs tid 61 v 605) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:41:35.486196 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 prepare_update pg_stats(0 pgs tid 61 v 605) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:41:35.486203 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 prepare_pg_stats pg_stats(0 pgs tid 61 v 605) v1 from osd.4 2013-04-03 10:41:35.486209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 got osd.4 osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[])) 2013-04-03 10:41:35.536389 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 encode_pending v 29067 2013-04-03 10:41:35.567142 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 update_from_paxos applying incremental 29067 2013-04-03 10:41:35.567153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 v29067: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1844B/s wr, 0op/s 2013-04-03 10:41:35.567170 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 send_pg_creates to 0 pgs 2013-04-03 10:41:35.567176 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 update_logger 2013-04-03 10:41:35.567188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 create_pending v 29068 2013-04-03 10:41:35.567200 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 _updated_stats for osd.4 10.252.24.32:6800/13473 2013-04-03 10:41:35.567217 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 check_osd_map already seen 605 >= 605 2013-04-03 10:41:35.567220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 update_logger 2013-04-03 10:41:35.567231 7fcdffcdb700 0 log [INF] : pgmap v29067: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1844B/s wr, 0op/s 2013-04-03 10:41:35.567290 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos 2013-04-03 10:41:35.567309 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos version 394065 summary v 394065 2013-04-03 10:41:35.567328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:35.567371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos 2013-04-03 10:41:35.567386 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos version 394065 summary v 394065 2013-04-03 10:41:35.567396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:35.567408 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:41:35.567433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:35.567449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:41:35.567460 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 logging 2013-04-03 10:41:35.567232 mon.0 10.252.24.25:6789/0 133 : [INF] pgmap v29067: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1844B/s wr, 0op/s 2013-04-03 10:41:35.580438 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos 2013-04-03 10:41:35.580455 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos version 394065 summary v 394065 2013-04-03 10:41:35.580472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:35.589652 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140407 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:36.566950 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 encode_pending v394066 2013-04-03 10:41:36.596235 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos 2013-04-03 10:41:36.596263 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos version 394066 summary v 394065 2013-04-03 10:41:36.596278 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos latest full 393798 2013-04-03 10:41:36.596296 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos applying incremental log 394066 2013-04-03 10:41:35.567232 mon.0 10.252.24.25:6789/0 133 : [INF] pgmap v29067: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1844B/s wr, 0op/s 2013-04-03 10:41:36.596349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 check_subs 2013-04-03 10:41:36.596380 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 create_pending v 394067 2013-04-03 10:41:36.596404 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:41:36.596471 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos 2013-04-03 10:41:36.596488 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos version 394066 summary v 394066 2013-04-03 10:41:36.596506 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:36.621947 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos 2013-04-03 10:41:36.621964 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos version 394066 summary v 394066 2013-04-03 10:41:36.621981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:36.830822 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:36.830832 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:41:36.830863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 preprocess_query pg_stats(0 pgs tid 410 v 605) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:41:36.830885 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 prepare_update pg_stats(0 pgs tid 410 v 605) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:41:36.830891 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 prepare_pg_stats pg_stats(0 pgs tid 410 v 605) v1 from osd.0 2013-04-03 10:41:36.830897 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 got osd.0 osd_stat(1140 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1139 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:41:36.935850 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:36.935858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a51b00 from 10.252.24.88:6789/0 2013-04-03 10:41:36.935889 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 122 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:41:37.388484 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21778 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:41:37.596066 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 encode_pending v 29068 2013-04-03 10:41:37.634209 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 update_from_paxos applying incremental 29068 2013-04-03 10:41:37.634221 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 v29068: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:41:37.634241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 send_pg_creates to 0 pgs 2013-04-03 10:41:37.634246 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 update_logger 2013-04-03 10:41:37.634257 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 create_pending v 29069 2013-04-03 10:41:37.634266 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:41:37.634284 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 check_osd_map already seen 605 >= 605 2013-04-03 10:41:37.634287 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 update_logger 2013-04-03 10:41:37.634303 7fcdffcdb700 0 log [INF] : pgmap v29068: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:41:37.634365 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos 2013-04-03 10:41:37.634385 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos version 394066 summary v 394066 2013-04-03 10:41:37.634404 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:37.634449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos 2013-04-03 10:41:37.634464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos version 394066 summary v 394066 2013-04-03 10:41:37.634474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:37.634486 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:41:37.634511 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:37.634525 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:41:37.635252 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 logging 2013-04-03 10:41:37.634305 mon.0 10.252.24.25:6789/0 134 : [INF] pgmap v29068: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:41:37.644062 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos 2013-04-03 10:41:37.644079 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos version 394066 summary v 394066 2013-04-03 10:41:37.644096 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:38.357041 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 check_down_pgs 2013-04-03 10:41:38.357078 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 v29068: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:41:38.357106 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:41:38.357131 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:41:38.357169 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:41:38.357201 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos 2013-04-03 10:41:38.357220 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos version 394066 summary v 394066 2013-04-03 10:41:38.357230 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 log 2013-04-03 10:41:38.357247 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:38.357266 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:41:38.622192 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 encode_pending v394067 2013-04-03 10:41:38.651568 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos 2013-04-03 10:41:38.651596 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos version 394067 summary v 394066 2013-04-03 10:41:38.651612 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos latest full 393798 2013-04-03 10:41:38.651631 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos applying incremental log 394067 2013-04-03 10:41:37.634305 mon.0 10.252.24.25:6789/0 134 : [INF] pgmap v29068: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:41:38.651679 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 check_subs 2013-04-03 10:41:38.651711 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 create_pending v 394068 2013-04-03 10:41:38.651734 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:41:38.651799 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos 2013-04-03 10:41:38.651815 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos version 394067 summary v 394067 2013-04-03 10:41:38.651834 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:38.662915 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos 2013-04-03 10:41:38.662933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos version 394067 summary v 394067 2013-04-03 10:41:38.662948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:39.589821 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140408 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:40.936017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:40.936028 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a52600 from 10.252.24.88:6789/0 2013-04-03 10:41:40.936081 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 123 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:41:41.388583 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21779 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:41:41.603737 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:41.603747 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:41:41.603797 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 preprocess_query pg_stats(1 pgs tid 40 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:41:41.603821 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 prepare_update pg_stats(1 pgs tid 40 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:41:41.603829 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 prepare_pg_stats pg_stats(1 pgs tid 40 v 605) v1 from osd.1 2013-04-03 10:41:41.603836 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:41:41.652374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos 2013-04-03 10:41:41.652394 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos version 394067 summary v 394067 2013-04-03 10:41:41.652411 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:41.654014 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 encode_pending v 29069 2013-04-03 10:41:41.692415 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 update_from_paxos applying incremental 29069 2013-04-03 10:41:41.692430 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 v29069: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 669B/s wr, 0op/s 2013-04-03 10:41:41.692449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 send_pg_creates to 0 pgs 2013-04-03 10:41:41.692454 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 update_logger 2013-04-03 10:41:41.692464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 create_pending v 29070 2013-04-03 10:41:41.692475 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:41:41.692491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 check_osd_map already seen 605 >= 605 2013-04-03 10:41:41.692493 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 update_logger 2013-04-03 10:41:41.692504 7fcdffcdb700 0 log [INF] : pgmap v29069: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 669B/s wr, 0op/s 2013-04-03 10:41:41.692569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos 2013-04-03 10:41:41.692587 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos version 394067 summary v 394067 2013-04-03 10:41:41.692604 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:41.692645 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos 2013-04-03 10:41:41.692660 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos version 394067 summary v 394067 2013-04-03 10:41:41.692669 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:41.692679 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:41:41.692702 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:41.692714 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:41:41.692724 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 logging 2013-04-03 10:41:41.692512 mon.0 10.252.24.25:6789/0 135 : [INF] pgmap v29069: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 669B/s wr, 0op/s 2013-04-03 10:41:41.721610 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos 2013-04-03 10:41:41.721627 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos version 394067 summary v 394067 2013-04-03 10:41:41.721642 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:41.769174 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:41.769181 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:41:41.769209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 preprocess_query pg_stats(1 pgs tid 30452 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:41:41.769229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 prepare_update pg_stats(1 pgs tid 30452 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:41:41.769235 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 prepare_pg_stats pg_stats(1 pgs tid 30452 v 605) v1 from osd.3 2013-04-03 10:41:41.769241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:41:42.687316 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 encode_pending v 29070 2013-04-03 10:41:42.688510 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 encode_pending v394068 2013-04-03 10:41:42.713733 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 update_from_paxos applying incremental 29070 2013-04-03 10:41:42.713752 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 v29070: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1005B/s wr, 0op/s 2013-04-03 10:41:42.713772 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 send_pg_creates to 0 pgs 2013-04-03 10:41:42.713777 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 update_logger 2013-04-03 10:41:42.713787 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 create_pending v 29071 2013-04-03 10:41:42.713797 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:41:42.713821 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 check_osd_map already seen 605 >= 605 2013-04-03 10:41:42.713825 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 update_logger 2013-04-03 10:41:42.713842 7fcdffcdb700 0 log [INF] : pgmap v29070: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1005B/s wr, 0op/s 2013-04-03 10:41:42.740423 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos 2013-04-03 10:41:42.740449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394067 2013-04-03 10:41:42.740464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos latest full 393798 2013-04-03 10:41:42.740482 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos applying incremental log 394068 2013-04-03 10:41:41.692512 mon.0 10.252.24.25:6789/0 135 : [INF] pgmap v29069: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 669B/s wr, 0op/s 2013-04-03 10:41:42.740526 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 check_subs 2013-04-03 10:41:42.740556 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 create_pending v 394069 2013-04-03 10:41:42.741118 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:41:42.741157 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos 2013-04-03 10:41:42.741173 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 2013-04-03 10:41:42.741183 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:42.741195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:41:42.741216 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 nothing new 2013-04-03 10:41:42.741267 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos 2013-04-03 10:41:42.741283 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 2013-04-03 10:41:42.741301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:42.790501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos 2013-04-03 10:41:42.790519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 2013-04-03 10:41:42.790536 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:43.357417 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 check_down_pgs 2013-04-03 10:41:43.357448 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 v29070: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1005B/s wr, 0op/s 2013-04-03 10:41:43.357477 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:41:43.357502 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:41:43.357540 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:41:43.357571 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos 2013-04-03 10:41:43.357590 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 2013-04-03 10:41:43.357600 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 log 2013-04-03 10:41:43.357622 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:43.357641 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:41:43.589946 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140409 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:44.936223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:44.936234 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2996080 from 10.252.24.88:6789/0 2013-04-03 10:41:44.936287 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 124 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:41:45.388694 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21780 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:41:45.741338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos 2013-04-03 10:41:45.741359 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 2013-04-03 10:41:45.741377 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:47.590055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140410 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:48.247281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:48.247292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:41:48.247342 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 preprocess_query pg_stats(1 pgs tid 20 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:41:48.247369 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 prepare_update pg_stats(1 pgs tid 20 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:41:48.247373 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 prepare_pg_stats pg_stats(1 pgs tid 20 v 605) v1 from osd.2 2013-04-03 10:41:48.247380 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:41:48.297520 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 encode_pending v 29071 2013-04-03 10:41:48.326326 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 update_from_paxos applying incremental 29071 2013-04-03 10:41:48.326348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 v29071: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:41:48.326367 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 send_pg_creates to 0 pgs 2013-04-03 10:41:48.326371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 update_logger 2013-04-03 10:41:48.326382 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 create_pending v 29072 2013-04-03 10:41:48.326392 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:41:48.326410 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 check_osd_map already seen 605 >= 605 2013-04-03 10:41:48.326413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 update_logger 2013-04-03 10:41:48.326424 7fcdffcdb700 0 log [INF] : pgmap v29071: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:41:48.326497 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos 2013-04-03 10:41:48.326518 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 2013-04-03 10:41:48.326537 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:48.326580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos 2013-04-03 10:41:48.326596 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 2013-04-03 10:41:48.326605 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:48.326617 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:41:48.326649 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:48.326662 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:41:48.326674 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 logging 2013-04-03 10:41:42.713845 mon.0 10.252.24.25:6789/0 136 : [INF] pgmap v29070: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1005B/s wr, 0op/s 2013-04-03 10:41:48.326696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 logging 2013-04-03 10:41:48.326425 mon.0 10.252.24.25:6789/0 137 : [INF] pgmap v29071: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:41:48.338229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos 2013-04-03 10:41:48.338246 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 2013-04-03 10:41:48.338263 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:48.357758 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 check_down_pgs 2013-04-03 10:41:48.357797 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 v29071: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:41:48.357821 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:41:48.357844 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:41:48.357882 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:41:48.357913 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos 2013-04-03 10:41:48.357930 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 2013-04-03 10:41:48.357940 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 log 2013-04-03 10:41:48.357956 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:48.357973 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:41:48.936271 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:48.936282 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2996b80 from 10.252.24.88:6789/0 2013-04-03 10:41:48.936334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 125 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:41:49.326076 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 encode_pending v394069 2013-04-03 10:41:49.357706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos 2013-04-03 10:41:49.357733 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394068 2013-04-03 10:41:49.357751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos latest full 393798 2013-04-03 10:41:49.357770 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos applying incremental log 394069 2013-04-03 10:41:42.713845 mon.0 10.252.24.25:6789/0 136 : [INF] pgmap v29070: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1005B/s wr, 0op/s 2013-04-03 10:41:49.357799 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos applying incremental log 394069 2013-04-03 10:41:48.326425 mon.0 10.252.24.25:6789/0 137 : [INF] pgmap v29071: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:41:49.357844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 check_subs 2013-04-03 10:41:49.358218 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 create_pending v 394070 2013-04-03 10:41:49.358245 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:41:49.358313 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos 2013-04-03 10:41:49.358330 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 2013-04-03 10:41:49.358349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:49.382523 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos 2013-04-03 10:41:49.382540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 2013-04-03 10:41:49.382556 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:49.388731 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21781 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:41:50.809217 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33093/0) 2013-04-03 10:41:51.590204 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140411 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:52.358580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos 2013-04-03 10:41:52.358601 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 2013-04-03 10:41:52.358619 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:52.936374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:52.936387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29952c0 from 10.252.24.88:6789/0 2013-04-03 10:41:52.936433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 126 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:41:53.358124 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 v29071: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:41:53.358170 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:41:53.358193 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:41:53.358235 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:41:53.358266 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos 2013-04-03 10:41:53.358286 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 2013-04-03 10:41:53.358295 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 log 2013-04-03 10:41:53.358312 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:53.358332 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:41:53.388880 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21782 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:41:55.358716 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos 2013-04-03 10:41:55.358738 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 2013-04-03 10:41:55.358755 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:55.590338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140412 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:56.605609 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:56.605620 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:41:56.605672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 preprocess_query pg_stats(1 pgs tid 41 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:41:56.605697 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 prepare_update pg_stats(1 pgs tid 41 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:41:56.605703 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 prepare_pg_stats pg_stats(1 pgs tid 41 v 605) v1 from osd.1 2013-04-03 10:41:56.605710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:41:56.656240 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 encode_pending v 29072 2013-04-03 10:41:56.684654 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 update_from_paxos applying incremental 29072 2013-04-03 10:41:56.684671 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 v29072: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s 2013-04-03 10:41:56.684691 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 send_pg_creates to 0 pgs 2013-04-03 10:41:56.684696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 update_logger 2013-04-03 10:41:56.684706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 create_pending v 29073 2013-04-03 10:41:56.684716 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:41:56.684731 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 check_osd_map already seen 605 >= 605 2013-04-03 10:41:56.684733 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 update_logger 2013-04-03 10:41:56.684744 7fcdffcdb700 0 log [INF] : pgmap v29072: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s 2013-04-03 10:41:56.684813 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos 2013-04-03 10:41:56.684834 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 2013-04-03 10:41:56.684853 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:56.684900 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos 2013-04-03 10:41:56.684916 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 2013-04-03 10:41:56.684925 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:56.684938 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:41:56.684963 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:56.684976 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:41:56.684988 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 logging 2013-04-03 10:41:56.684752 mon.0 10.252.24.25:6789/0 138 : [INF] pgmap v29072: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s 2013-04-03 10:41:56.696019 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos 2013-04-03 10:41:56.696040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 2013-04-03 10:41:56.696058 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:56.936456 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:56.936465 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83840 from 10.252.24.88:6789/0 2013-04-03 10:41:56.936499 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 127 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:41:57.389041 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21783 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:41:57.684065 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 encode_pending v394070 2013-04-03 10:41:57.712738 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos 2013-04-03 10:41:57.712776 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos version 394070 summary v 394069 2013-04-03 10:41:57.712791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos latest full 393798 2013-04-03 10:41:57.712809 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos applying incremental log 394070 2013-04-03 10:41:56.684752 mon.0 10.252.24.25:6789/0 138 : [INF] pgmap v29072: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s 2013-04-03 10:41:57.712858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 check_subs 2013-04-03 10:41:57.712889 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 create_pending v 394071 2013-04-03 10:41:57.712913 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:41:57.712983 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos 2013-04-03 10:41:57.712999 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos version 394070 summary v 394070 2013-04-03 10:41:57.713018 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:57.737154 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos 2013-04-03 10:41:57.737171 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos version 394070 summary v 394070 2013-04-03 10:41:57.737186 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:58.248654 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:41:58.248663 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:41:58.248696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 preprocess_query pg_stats(1 pgs tid 21 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:41:58.248718 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 prepare_update pg_stats(1 pgs tid 21 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:41:58.248724 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 prepare_pg_stats pg_stats(1 pgs tid 21 v 605) v1 from osd.2 2013-04-03 10:41:58.248731 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:41:58.358474 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 check_down_pgs 2013-04-03 10:41:58.358510 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 v29072: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s 2013-04-03 10:41:58.358545 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:41:58.358571 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:41:58.358608 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:41:58.358640 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos 2013-04-03 10:41:58.358659 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos version 394070 summary v 394070 2013-04-03 10:41:58.358669 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 log 2013-04-03 10:41:58.358686 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:41:58.358705 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:41:58.597490 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) 2013-04-03 10:41:58.597520 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader 2013-04-03 10:41:58.712648 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29072 encode_pending v 29073 2013-04-03 10:41:58.745586 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29072 update_from_paxos applying incremental 29073 2013-04-03 10:41:58.745606 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 v29073: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 7078B/s wr, 0op/s 2013-04-03 10:41:58.745624 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 send_pg_creates to 0 pgs 2013-04-03 10:41:58.745626 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 update_logger 2013-04-03 10:41:58.745636 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 create_pending v 29074 2013-04-03 10:41:58.745644 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29073 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:41:58.745662 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 check_osd_map already seen 605 >= 605 2013-04-03 10:41:58.745665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 update_logger 2013-04-03 10:41:58.745675 7fcdffcdb700 0 log [INF] : pgmap v29073: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 7078B/s wr, 0op/s 2013-04-03 10:41:58.745737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 update_from_paxos 2013-04-03 10:41:58.745756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 update_from_paxos version 394070 summary v 394070 2013-04-03 10:41:58.745775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:58.745830 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 update_from_paxos 2013-04-03 10:41:58.745846 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 update_from_paxos version 394070 summary v 394070 2013-04-03 10:41:58.745855 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:58.745867 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:41:58.745891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:41:58.745903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:41:58.745914 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 logging 2013-04-03 10:41:58.745676 mon.0 10.252.24.25:6789/0 139 : [INF] pgmap v29073: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 7078B/s wr, 0op/s 2013-04-03 10:41:58.756780 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 update_from_paxos 2013-04-03 10:41:58.756796 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 update_from_paxos version 394070 summary v 394070 2013-04-03 10:41:58.756812 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:59.590458 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140413 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:41:59.745312 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394070 encode_pending v394071 2013-04-03 10:41:59.773678 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos 2013-04-03 10:41:59.773705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394070 2013-04-03 10:41:59.773724 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos latest full 393798 2013-04-03 10:41:59.773742 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394071 update_from_paxos applying incremental log 394071 2013-04-03 10:41:58.745676 mon.0 10.252.24.25:6789/0 139 : [INF] pgmap v29073: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 7078B/s wr, 0op/s 2013-04-03 10:41:59.773791 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 check_subs 2013-04-03 10:41:59.773822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 create_pending v 394072 2013-04-03 10:41:59.773844 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394071 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:41:59.773908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos 2013-04-03 10:41:59.773924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394071 2013-04-03 10:41:59.773942 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:41:59.784614 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos 2013-04-03 10:41:59.784631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394071 2013-04-03 10:41:59.784647 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:00.936624 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:00.936633 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a512c0 from 10.252.24.88:6789/0 2013-04-03 10:42:00.936688 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 128 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:42:01.389141 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21784 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:42:02.774697 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos 2013-04-03 10:42:02.774717 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394071 2013-04-03 10:42:02.774734 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:03.249558 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:03.249568 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:42:03.249599 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 preprocess_query pg_stats(1 pgs tid 22 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:42:03.249620 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 prepare_update pg_stats(1 pgs tid 22 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:42:03.249626 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 prepare_pg_stats pg_stats(1 pgs tid 22 v 605) v1 from osd.2 2013-04-03 10:42:03.249632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:42:03.299816 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29073 encode_pending v 29074 2013-04-03 10:42:03.358855 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:42:03.358894 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:42:03.358917 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos 2013-04-03 10:42:03.358936 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394071 2013-04-03 10:42:03.358945 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394071 log 2013-04-03 10:42:03.358962 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:03.358981 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:42:03.365191 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29073 update_from_paxos applying incremental 29074 2013-04-03 10:42:03.365211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 v29074: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 11097B/s wr, 0op/s 2013-04-03 10:42:03.365230 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 send_pg_creates to 0 pgs 2013-04-03 10:42:03.365234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 update_logger 2013-04-03 10:42:03.365244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 create_pending v 29075 2013-04-03 10:42:03.365253 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29074 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:42:03.365270 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 check_osd_map already seen 605 >= 605 2013-04-03 10:42:03.365272 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 update_logger 2013-04-03 10:42:03.365283 7fcdffcdb700 0 log [INF] : pgmap v29074: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 11097B/s wr, 0op/s 2013-04-03 10:42:03.365343 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos 2013-04-03 10:42:03.365360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394071 2013-04-03 10:42:03.365377 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:03.365419 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos 2013-04-03 10:42:03.365436 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394071 2013-04-03 10:42:03.365445 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:03.365799 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:42:03.365839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:03.365854 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:42:03.365865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 logging 2013-04-03 10:42:03.365284 mon.0 10.252.24.25:6789/0 140 : [INF] pgmap v29074: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 11097B/s wr, 0op/s 2013-04-03 10:42:03.377932 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos 2013-04-03 10:42:03.377948 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394071 2013-04-03 10:42:03.377963 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:03.590521 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140414 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:42:04.364927 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394071 encode_pending v394072 2013-04-03 10:42:04.410102 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos 2013-04-03 10:42:04.410129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394071 2013-04-03 10:42:04.410143 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos latest full 393798 2013-04-03 10:42:04.410164 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394072 update_from_paxos applying incremental log 394072 2013-04-03 10:42:03.365284 mon.0 10.252.24.25:6789/0 140 : [INF] pgmap v29074: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 11097B/s wr, 0op/s 2013-04-03 10:42:04.410209 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 check_subs 2013-04-03 10:42:04.410240 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 create_pending v 394073 2013-04-03 10:42:04.410262 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394072 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:42:04.410327 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos 2013-04-03 10:42:04.410342 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 2013-04-03 10:42:04.410360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:04.423000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos 2013-04-03 10:42:04.423016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 2013-04-03 10:42:04.423031 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:04.936763 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:04.936772 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52080 from 10.252.24.88:6789/0 2013-04-03 10:42:04.936822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 129 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:42:05.389227 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21785 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:42:07.410985 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos 2013-04-03 10:42:07.411005 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 2013-04-03 10:42:07.411022 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:07.590641 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140415 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:42:08.359101 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29074 check_down_pgs 2013-04-03 10:42:08.359130 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29074 v29074: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 11097B/s wr, 0op/s 2013-04-03 10:42:08.359158 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:42:08.359187 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:42:08.359223 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:42:08.359254 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos 2013-04-03 10:42:08.359273 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 2013-04-03 10:42:08.359282 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394072 log 2013-04-03 10:42:08.359299 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:08.359318 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:42:08.936928 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:08.936937 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a51000 from 10.252.24.88:6789/0 2013-04-03 10:42:08.936989 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 130 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:42:09.389347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21786 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:42:10.411087 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos 2013-04-03 10:42:10.411108 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 2013-04-03 10:42:10.411143 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:11.590825 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140416 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:42:11.607266 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:11.607275 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:42:11.607307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 preprocess_query pg_stats(1 pgs tid 42 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:42:11.607331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 prepare_update pg_stats(1 pgs tid 42 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:42:11.607337 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 prepare_pg_stats pg_stats(1 pgs tid 42 v 605) v1 from osd.1 2013-04-03 10:42:11.607343 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:42:11.657528 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29074 encode_pending v 29075 2013-04-03 10:42:11.691398 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29074 update_from_paxos applying incremental 29075 2013-04-03 10:42:11.691417 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 v29075: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 632B/s wr, 0op/s 2013-04-03 10:42:11.691438 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 send_pg_creates to 0 pgs 2013-04-03 10:42:11.691443 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 update_logger 2013-04-03 10:42:11.691453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 create_pending v 29076 2013-04-03 10:42:11.691461 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29075 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:42:11.691478 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 check_osd_map already seen 605 >= 605 2013-04-03 10:42:11.691481 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 update_logger 2013-04-03 10:42:11.691492 7fcdffcdb700 0 log [INF] : pgmap v29075: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 632B/s wr, 0op/s 2013-04-03 10:42:11.691647 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos 2013-04-03 10:42:11.691669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 2013-04-03 10:42:11.691688 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:11.691732 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos 2013-04-03 10:42:11.691746 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 2013-04-03 10:42:11.691755 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:11.691769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:42:11.691792 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:11.691804 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:42:11.691813 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 logging 2013-04-03 10:42:11.691493 mon.0 10.252.24.25:6789/0 141 : [INF] pgmap v29075: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 632B/s wr, 0op/s 2013-04-03 10:42:11.702948 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos 2013-04-03 10:42:11.702965 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 2013-04-03 10:42:11.702979 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:11.772247 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:11.772254 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:42:11.772281 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 preprocess_query pg_stats(1 pgs tid 30453 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:42:11.772301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 prepare_update pg_stats(1 pgs tid 30453 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:42:11.772306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 prepare_pg_stats pg_stats(1 pgs tid 30453 v 605) v1 from osd.3 2013-04-03 10:42:11.772310 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:42:12.691148 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394072 encode_pending v394073 2013-04-03 10:42:12.692683 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29075 encode_pending v 29076 2013-04-03 10:42:12.719167 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos 2013-04-03 10:42:12.719196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos version 394073 summary v 394072 2013-04-03 10:42:12.719212 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos latest full 393798 2013-04-03 10:42:12.719231 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394073 update_from_paxos applying incremental log 394073 2013-04-03 10:42:11.691493 mon.0 10.252.24.25:6789/0 141 : [INF] pgmap v29075: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 632B/s wr, 0op/s 2013-04-03 10:42:12.719407 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 check_subs 2013-04-03 10:42:12.719441 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 create_pending v 394074 2013-04-03 10:42:12.719463 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394073 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:42:12.744797 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29075 update_from_paxos applying incremental 29076 2013-04-03 10:42:12.744820 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 v29076: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:42:12.744843 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 send_pg_creates to 0 pgs 2013-04-03 10:42:12.744848 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 update_logger 2013-04-03 10:42:12.744859 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 create_pending v 29077 2013-04-03 10:42:12.744868 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29076 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:42:12.744885 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 check_osd_map already seen 605 >= 605 2013-04-03 10:42:12.744888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 update_logger 2013-04-03 10:42:12.744898 7fcdffcdb700 0 log [INF] : pgmap v29076: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:42:12.744957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos 2013-04-03 10:42:12.744974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos version 394073 summary v 394073 2013-04-03 10:42:12.744992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:12.745033 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos 2013-04-03 10:42:12.745048 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos version 394073 summary v 394073 2013-04-03 10:42:12.745057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:12.745069 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:42:12.745092 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:12.745105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:42:12.745116 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 logging 2013-04-03 10:42:12.744899 mon.0 10.252.24.25:6789/0 142 : [INF] pgmap v29076: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:42:12.769424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos 2013-04-03 10:42:12.769440 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos version 394073 summary v 394073 2013-04-03 10:42:12.769455 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:12.937034 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:12.937043 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4340 from 10.252.24.88:6789/0 2013-04-03 10:42:12.937093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 131 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:42:13.250789 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:13.250796 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:42:13.250829 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 preprocess_query pg_stats(1 pgs tid 23 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:42:13.250853 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 prepare_update pg_stats(1 pgs tid 23 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:42:13.250859 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 prepare_pg_stats pg_stats(1 pgs tid 23 v 605) v1 from osd.2 2013-04-03 10:42:13.250865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:42:13.359469 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29076 check_down_pgs 2013-04-03 10:42:13.359499 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29076 v29076: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:42:13.359526 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:42:13.359551 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:42:13.359587 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:42:13.359618 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos 2013-04-03 10:42:13.359636 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos version 394073 summary v 394073 2013-04-03 10:42:13.359645 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394073 log 2013-04-03 10:42:13.359662 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:13.359681 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:42:13.389423 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21787 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:42:13.744106 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29076 encode_pending v 29077 2013-04-03 10:42:13.745641 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394073 encode_pending v394074 2013-04-03 10:42:13.774807 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29076 update_from_paxos applying incremental 29077 2013-04-03 10:42:13.774826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 v29077: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 12268B/s wr, 0op/s 2013-04-03 10:42:13.774844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 send_pg_creates to 0 pgs 2013-04-03 10:42:13.774847 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 update_logger 2013-04-03 10:42:13.774856 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 create_pending v 29078 2013-04-03 10:42:13.774864 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29077 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:42:13.774881 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 check_osd_map already seen 605 >= 605 2013-04-03 10:42:13.774885 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 update_logger 2013-04-03 10:42:13.774895 7fcdffcdb700 0 log [INF] : pgmap v29077: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 12268B/s wr, 0op/s 2013-04-03 10:42:13.806288 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos 2013-04-03 10:42:13.806312 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394073 2013-04-03 10:42:13.806326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos latest full 393798 2013-04-03 10:42:13.806342 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394074 update_from_paxos applying incremental log 394074 2013-04-03 10:42:12.744899 mon.0 10.252.24.25:6789/0 142 : [INF] pgmap v29076: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:42:13.806386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 check_subs 2013-04-03 10:42:13.806416 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 create_pending v 394075 2013-04-03 10:42:13.806438 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394074 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:42:13.806471 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos 2013-04-03 10:42:13.806486 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 2013-04-03 10:42:13.806495 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:13.806508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:42:13.806526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 nothing new 2013-04-03 10:42:13.806570 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos 2013-04-03 10:42:13.806585 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 2013-04-03 10:42:13.806730 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:13.830519 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos 2013-04-03 10:42:13.830536 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 2013-04-03 10:42:13.830550 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:15.590948 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140417 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:42:16.807188 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos 2013-04-03 10:42:16.807209 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 2013-04-03 10:42:16.807226 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:16.937156 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:16.937165 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e48c0 from 10.252.24.88:6789/0 2013-04-03 10:42:16.937218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 132 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:42:17.389535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21788 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:42:18.359832 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29077 check_down_pgs 2013-04-03 10:42:18.359866 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29077 v29077: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 12268B/s wr, 0op/s 2013-04-03 10:42:18.359894 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:42:18.359918 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:42:18.359954 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:42:18.359985 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos 2013-04-03 10:42:18.360004 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 2013-04-03 10:42:18.360013 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394074 log 2013-04-03 10:42:18.360030 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:18.360049 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:42:19.591056 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140418 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:42:19.807234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos 2013-04-03 10:42:19.807254 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 2013-04-03 10:42:19.807271 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:20.937284 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:20.937294 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4b80 from 10.252.24.88:6789/0 2013-04-03 10:42:20.937355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 133 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:42:21.389651 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21789 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:42:22.807336 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos 2013-04-03 10:42:22.807356 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 2013-04-03 10:42:22.807373 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:23.360196 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29077 v29077: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 12268B/s wr, 0op/s 2013-04-03 10:42:23.360237 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:42:23.360260 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:42:23.360305 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:42:23.360335 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos 2013-04-03 10:42:23.360354 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 2013-04-03 10:42:23.360363 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394074 log 2013-04-03 10:42:23.360514 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:23.360538 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:42:23.591142 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140419 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:42:24.937514 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:24.937524 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4080 from 10.252.24.88:6789/0 2013-04-03 10:42:24.937585 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 134 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:42:25.389736 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21790 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:42:25.807326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos 2013-04-03 10:42:25.807346 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 2013-04-03 10:42:25.807363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:26.608954 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:26.608964 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:42:26.609016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 preprocess_query pg_stats(1 pgs tid 43 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:42:26.609041 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 prepare_update pg_stats(1 pgs tid 43 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:42:26.609046 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 prepare_pg_stats pg_stats(1 pgs tid 43 v 605) v1 from osd.1 2013-04-03 10:42:26.609052 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:42:26.659236 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29077 encode_pending v 29078 2013-04-03 10:42:26.687309 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29077 update_from_paxos applying incremental 29078 2013-04-03 10:42:26.687324 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29078 v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s 2013-04-03 10:42:26.687344 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29078 send_pg_creates to 0 pgs 2013-04-03 10:42:26.687347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29078 update_logger 2013-04-03 10:42:26.687357 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29078 create_pending v 29079 2013-04-03 10:42:26.687372 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29078 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:42:26.687389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29078 check_osd_map already seen 605 >= 605 2013-04-03 10:42:26.687392 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29078 update_logger 2013-04-03 10:42:26.687402 7fcdffcdb700 0 log [INF] : pgmap v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s 2013-04-03 10:42:26.687466 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos 2013-04-03 10:42:26.687485 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 2013-04-03 10:42:26.687504 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:26.687546 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos 2013-04-03 10:42:26.687560 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 2013-04-03 10:42:26.687569 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:26.687585 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:42:26.687617 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:26.687632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:42:26.687643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 logging 2013-04-03 10:42:13.774896 mon.0 10.252.24.25:6789/0 143 : [INF] pgmap v29077: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 12268B/s wr, 0op/s 2013-04-03 10:42:26.687664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 logging 2013-04-03 10:42:26.687403 mon.0 10.252.24.25:6789/0 144 : [INF] pgmap v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s 2013-04-03 10:42:26.698937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos 2013-04-03 10:42:26.698955 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 2013-04-03 10:42:26.698971 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:27.591297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140420 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:42:27.687042 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394074 encode_pending v394075 2013-04-03 10:42:27.740266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos 2013-04-03 10:42:27.740293 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos version 394075 summary v 394074 2013-04-03 10:42:27.740307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos latest full 393798 2013-04-03 10:42:27.740326 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394075 update_from_paxos applying incremental log 394075 2013-04-03 10:42:13.774896 mon.0 10.252.24.25:6789/0 143 : [INF] pgmap v29077: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 12268B/s wr, 0op/s 2013-04-03 10:42:27.740356 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394075 update_from_paxos applying incremental log 394075 2013-04-03 10:42:26.687403 mon.0 10.252.24.25:6789/0 144 : [INF] pgmap v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s 2013-04-03 10:42:27.740405 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 check_subs 2013-04-03 10:42:27.740439 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 create_pending v 394076 2013-04-03 10:42:27.740461 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394075 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:42:27.740524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos 2013-04-03 10:42:27.740539 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos version 394075 summary v 394075 2013-04-03 10:42:27.740558 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:27.765232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos 2013-04-03 10:42:27.765249 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos version 394075 summary v 394075 2013-04-03 10:42:27.765266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:28.360688 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29078 check_down_pgs 2013-04-03 10:42:28.360718 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29078 v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s 2013-04-03 10:42:28.360746 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:42:28.360770 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:42:28.360806 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:42:28.360837 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos 2013-04-03 10:42:28.360856 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos version 394075 summary v 394075 2013-04-03 10:42:28.360865 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394075 log 2013-04-03 10:42:28.360882 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:28.360905 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:42:28.694893 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:42:28.694899 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:42:28.694910 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 2013-04-03 10:42:28.937483 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:28.937493 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ce2340 from 10.252.24.88:6789/0 2013-04-03 10:42:28.937536 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 135 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:42:29.389894 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21791 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:42:30.741134 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos 2013-04-03 10:42:30.741155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 2013-04-03 10:42:30.741173 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:31.591429 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140421 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:42:32.937773 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:32.937784 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ce1840 from 10.252.24.88:6789/0 2013-04-03 10:42:32.937839 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 136 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:42:33.361055 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s 2013-04-03 10:42:33.361097 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:42:33.361121 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:42:33.361173 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:42:33.361206 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos 2013-04-03 10:42:33.361225 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 2013-04-03 10:42:33.361235 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 log 2013-04-03 10:42:33.361252 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:33.361271 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:42:33.389948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21792 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:42:33.741231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos 2013-04-03 10:42:33.741249 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 2013-04-03 10:42:33.741264 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:35.591525 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140422 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:42:36.741340 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos 2013-04-03 10:42:36.741361 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 2013-04-03 10:42:36.741380 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:36.937782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:36.937792 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ce1000 from 10.252.24.88:6789/0 2013-04-03 10:42:36.937824 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 137 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:42:37.390092 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21793 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:42:38.361420 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s 2013-04-03 10:42:38.361461 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:42:38.361485 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:42:38.361531 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:42:38.361563 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos 2013-04-03 10:42:38.361587 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 2013-04-03 10:42:38.361597 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 log 2013-04-03 10:42:38.361614 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:38.361633 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:42:39.591769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140423 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:42:39.741352 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos 2013-04-03 10:42:39.741372 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 2013-04-03 10:42:39.741390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:40.937934 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:40.937945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273b600 from 10.252.24.88:6789/0 2013-04-03 10:42:40.937999 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 138 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:42:41.390229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21794 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:42:42.741454 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos 2013-04-03 10:42:42.741475 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 2013-04-03 10:42:42.741493 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:43.254614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:43.254625 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:42:43.254676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 preprocess_query pg_stats(1 pgs tid 24 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:42:43.254701 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 prepare_update pg_stats(1 pgs tid 24 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:42:43.254706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 prepare_pg_stats pg_stats(1 pgs tid 24 v 605) v1 from osd.2 2013-04-03 10:42:43.254712 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:42:43.304895 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 encode_pending v 29079 2013-04-03 10:42:43.361753 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:42:43.361790 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:42:43.361814 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos 2013-04-03 10:42:43.361834 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 2013-04-03 10:42:43.361843 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 log 2013-04-03 10:42:43.362491 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:43.362519 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:42:43.387713 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 update_from_paxos applying incremental 29079 2013-04-03 10:42:43.387734 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 v29079: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3186B/s wr, 0op/s 2013-04-03 10:42:43.387754 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 send_pg_creates to 0 pgs 2013-04-03 10:42:43.387759 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 update_logger 2013-04-03 10:42:43.387769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 create_pending v 29080 2013-04-03 10:42:43.387779 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:42:43.387800 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 check_osd_map already seen 605 >= 605 2013-04-03 10:42:43.387804 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 update_logger 2013-04-03 10:42:43.387814 7fcdffcdb700 0 log [INF] : pgmap v29079: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3186B/s wr, 0op/s 2013-04-03 10:42:43.387873 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos 2013-04-03 10:42:43.387891 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 2013-04-03 10:42:43.387912 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:43.387958 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos 2013-04-03 10:42:43.387974 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 2013-04-03 10:42:43.387983 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:43.387995 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:42:43.388021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:43.388034 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:42:43.388044 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 logging 2013-04-03 10:42:43.387816 mon.0 10.252.24.25:6789/0 145 : [INF] pgmap v29079: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3186B/s wr, 0op/s 2013-04-03 10:42:43.399302 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos 2013-04-03 10:42:43.399319 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 2013-04-03 10:42:43.399336 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:43.591729 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140424 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:42:44.387499 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 encode_pending v394076 2013-04-03 10:42:44.415589 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos 2013-04-03 10:42:44.415617 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394075 2013-04-03 10:42:44.415637 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos latest full 393798 2013-04-03 10:42:44.415656 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos applying incremental log 394076 2013-04-03 10:42:43.387816 mon.0 10.252.24.25:6789/0 145 : [INF] pgmap v29079: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3186B/s wr, 0op/s 2013-04-03 10:42:44.415706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 check_subs 2013-04-03 10:42:44.415739 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 create_pending v 394077 2013-04-03 10:42:44.415762 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:42:44.415835 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos 2013-04-03 10:42:44.415851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 2013-04-03 10:42:44.415870 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:44.440442 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos 2013-04-03 10:42:44.440459 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 2013-04-03 10:42:44.440474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:44.938033 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:44.938043 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273adc0 from 10.252.24.88:6789/0 2013-04-03 10:42:44.938079 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 139 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:42:45.390336 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21795 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:42:47.416580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos 2013-04-03 10:42:47.416728 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 2013-04-03 10:42:47.416748 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:47.591849 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140425 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:42:48.362640 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 check_down_pgs 2013-04-03 10:42:48.362670 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 v29079: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3186B/s wr, 0op/s 2013-04-03 10:42:48.362698 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:42:48.362724 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:42:48.362761 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:42:48.362793 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos 2013-04-03 10:42:48.362812 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 2013-04-03 10:42:48.362822 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 log 2013-04-03 10:42:48.362839 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:48.362859 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:42:48.938163 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:48.938174 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2996340 from 10.252.24.88:6789/0 2013-04-03 10:42:48.938220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 140 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:42:49.390476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21796 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:42:50.416669 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos 2013-04-03 10:42:50.416896 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 2013-04-03 10:42:50.416916 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:51.592008 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140426 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:42:51.611594 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:51.611603 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:42:51.611637 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 preprocess_query pg_stats(1 pgs tid 44 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:42:51.611662 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 prepare_update pg_stats(1 pgs tid 44 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:42:51.611668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 prepare_pg_stats pg_stats(1 pgs tid 44 v 605) v1 from osd.1 2013-04-03 10:42:51.611675 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:42:51.661858 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 encode_pending v 29080 2013-04-03 10:42:51.692024 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 update_from_paxos applying incremental 29080 2013-04-03 10:42:51.692044 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 v29080: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3767B/s wr, 0op/s 2013-04-03 10:42:51.692066 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 send_pg_creates to 0 pgs 2013-04-03 10:42:51.692070 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 update_logger 2013-04-03 10:42:51.692079 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 create_pending v 29081 2013-04-03 10:42:51.692089 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:42:51.692105 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 check_osd_map already seen 605 >= 605 2013-04-03 10:42:51.692109 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 update_logger 2013-04-03 10:42:51.692120 7fcdffcdb700 0 log [INF] : pgmap v29080: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3767B/s wr, 0op/s 2013-04-03 10:42:51.692175 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos 2013-04-03 10:42:51.692195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 2013-04-03 10:42:51.692215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:51.692258 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos 2013-04-03 10:42:51.692274 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 2013-04-03 10:42:51.692286 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:51.692297 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:42:51.692322 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:51.692335 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:42:51.692345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 logging 2013-04-03 10:42:51.692121 mon.0 10.252.24.25:6789/0 146 : [INF] pgmap v29080: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3767B/s wr, 0op/s 2013-04-03 10:42:51.704825 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos 2013-04-03 10:42:51.704843 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 2013-04-03 10:42:51.704858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:51.776952 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:51.776959 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:42:51.776987 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 preprocess_query pg_stats(1 pgs tid 30454 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:42:51.777009 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 prepare_update pg_stats(1 pgs tid 30454 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:42:51.777014 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 prepare_pg_stats pg_stats(1 pgs tid 30454 v 605) v1 from osd.3 2013-04-03 10:42:51.777019 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:42:52.691742 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 encode_pending v 29081 2013-04-03 10:42:52.692981 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 encode_pending v394077 2013-04-03 10:42:52.721273 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 update_from_paxos applying incremental 29081 2013-04-03 10:42:52.721293 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 v29081: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:42:52.721313 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 send_pg_creates to 0 pgs 2013-04-03 10:42:52.721317 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 update_logger 2013-04-03 10:42:52.721326 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 create_pending v 29082 2013-04-03 10:42:52.721336 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:42:52.721354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 check_osd_map already seen 605 >= 605 2013-04-03 10:42:52.721357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 update_logger 2013-04-03 10:42:52.721368 7fcdffcdb700 0 log [INF] : pgmap v29081: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:42:52.758267 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos 2013-04-03 10:42:52.758292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394076 2013-04-03 10:42:52.758309 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos latest full 393798 2013-04-03 10:42:52.758329 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos applying incremental log 394077 2013-04-03 10:42:51.692121 mon.0 10.252.24.25:6789/0 146 : [INF] pgmap v29080: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3767B/s wr, 0op/s 2013-04-03 10:42:52.758374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 check_subs 2013-04-03 10:42:52.758406 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 create_pending v 394078 2013-04-03 10:42:52.758429 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:42:52.758462 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos 2013-04-03 10:42:52.758477 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 2013-04-03 10:42:52.758487 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:52.758499 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:42:52.758519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 nothing new 2013-04-03 10:42:52.758568 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos 2013-04-03 10:42:52.758584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 2013-04-03 10:42:52.758603 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:52.770066 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos 2013-04-03 10:42:52.770083 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 2013-04-03 10:42:52.770099 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:52.938352 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:52.938363 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc4dc0 from 10.252.24.88:6789/0 2013-04-03 10:42:52.938417 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 141 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:42:53.362979 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 check_down_pgs 2013-04-03 10:42:53.363010 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 v29081: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:42:53.363040 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:42:53.363066 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:42:53.363104 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:42:53.363136 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos 2013-04-03 10:42:53.363157 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 2013-04-03 10:42:53.363167 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 log 2013-04-03 10:42:53.363185 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:53.363555 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:42:53.390555 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21797 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:42:55.592131 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140427 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:42:55.759124 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos 2013-04-03 10:42:55.759148 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 2013-04-03 10:42:55.759166 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:56.938386 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:56.938397 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc5340 from 10.252.24.88:6789/0 2013-04-03 10:42:56.938452 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 142 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:42:57.390716 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21798 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:42:58.256579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:42:58.256588 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:42:58.256626 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 preprocess_query pg_stats(1 pgs tid 25 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:42:58.256649 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 prepare_update pg_stats(1 pgs tid 25 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:42:58.256654 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 prepare_pg_stats pg_stats(1 pgs tid 25 v 605) v1 from osd.2 2013-04-03 10:42:58.256660 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:42:58.306849 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 encode_pending v 29082 2013-04-03 10:42:58.332737 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 update_from_paxos applying incremental 29082 2013-04-03 10:42:58.332754 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 v29082: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3236B/s wr, 0op/s 2013-04-03 10:42:58.332774 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 send_pg_creates to 0 pgs 2013-04-03 10:42:58.332779 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 update_logger 2013-04-03 10:42:58.332790 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 create_pending v 29083 2013-04-03 10:42:58.332801 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:42:58.332822 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 check_osd_map already seen 605 >= 605 2013-04-03 10:42:58.332826 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 update_logger 2013-04-03 10:42:58.332836 7fcdffcdb700 0 log [INF] : pgmap v29082: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3236B/s wr, 0op/s 2013-04-03 10:42:58.332898 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos 2013-04-03 10:42:58.332918 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 2013-04-03 10:42:58.332937 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:58.332985 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos 2013-04-03 10:42:58.333000 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 2013-04-03 10:42:58.333010 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:58.333021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:42:58.333053 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:42:58.333066 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:42:58.333076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 logging 2013-04-03 10:42:52.721369 mon.0 10.252.24.25:6789/0 147 : [INF] pgmap v29081: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:42:58.333097 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 logging 2013-04-03 10:42:58.332837 mon.0 10.252.24.25:6789/0 148 : [INF] pgmap v29082: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3236B/s wr, 0op/s 2013-04-03 10:42:58.356354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos 2013-04-03 10:42:58.356371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 2013-04-03 10:42:58.356387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:58.363662 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 check_down_pgs 2013-04-03 10:42:58.363688 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 v29082: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3236B/s wr, 0op/s 2013-04-03 10:42:58.363716 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:42:58.363741 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:42:58.363779 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:42:58.363810 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos 2013-04-03 10:42:58.363829 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 2013-04-03 10:42:58.363838 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 log 2013-04-03 10:42:58.363855 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:42:58.363874 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:42:58.694978 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) 2013-04-03 10:42:58.695018 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader 2013-04-03 10:42:59.332487 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394077 encode_pending v394078 2013-04-03 10:42:59.356149 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos 2013-04-03 10:42:59.356173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394077 2013-04-03 10:42:59.356187 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos latest full 393798 2013-04-03 10:42:59.356209 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394078 update_from_paxos applying incremental log 394078 2013-04-03 10:42:52.721369 mon.0 10.252.24.25:6789/0 147 : [INF] pgmap v29081: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:42:59.356237 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394078 update_from_paxos applying incremental log 394078 2013-04-03 10:42:58.332837 mon.0 10.252.24.25:6789/0 148 : [INF] pgmap v29082: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3236B/s wr, 0op/s 2013-04-03 10:42:59.356273 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 check_subs 2013-04-03 10:42:59.356304 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 create_pending v 394079 2013-04-03 10:42:59.356326 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394078 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:42:59.356391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos 2013-04-03 10:42:59.356406 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 2013-04-03 10:42:59.356423 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:59.368910 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos 2013-04-03 10:42:59.368929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 2013-04-03 10:42:59.368946 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:42:59.592322 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140428 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:43:00.938707 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:00.938716 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2146080 from 10.252.24.88:6789/0 2013-04-03 10:43:00.938769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 143 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:43:01.390813 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21799 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:43:02.357017 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos 2013-04-03 10:43:02.357037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 2013-04-03 10:43:02.357054 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:03.364012 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29082 v29082: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3236B/s wr, 0op/s 2013-04-03 10:43:03.364059 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:43:03.364080 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:43:03.364124 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:43:03.364156 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos 2013-04-03 10:43:03.364174 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 2013-04-03 10:43:03.364184 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394078 log 2013-04-03 10:43:03.364200 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:03.364219 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:43:03.592419 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140429 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:43:04.938683 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:04.938693 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2145840 from 10.252.24.88:6789/0 2013-04-03 10:43:04.938750 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 144 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:43:05.357015 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos 2013-04-03 10:43:05.357035 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 2013-04-03 10:43:05.357052 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:05.390917 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21800 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:43:06.613436 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:06.613446 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:43:06.613498 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29082 preprocess_query pg_stats(1 pgs tid 45 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:43:06.613523 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29082 prepare_update pg_stats(1 pgs tid 45 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:43:06.613529 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29082 prepare_pg_stats pg_stats(1 pgs tid 45 v 605) v1 from osd.1 2013-04-03 10:43:06.613535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29082 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:43:06.663720 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29082 encode_pending v 29083 2013-04-03 10:43:06.692840 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29082 update_from_paxos applying incremental 29083 2013-04-03 10:43:06.692855 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 v29083: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s 2013-04-03 10:43:06.692875 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 send_pg_creates to 0 pgs 2013-04-03 10:43:06.692877 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 update_logger 2013-04-03 10:43:06.692886 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 create_pending v 29084 2013-04-03 10:43:06.692895 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29083 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:43:06.692911 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 check_osd_map already seen 605 >= 605 2013-04-03 10:43:06.692914 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 update_logger 2013-04-03 10:43:06.692924 7fcdffcdb700 0 log [INF] : pgmap v29083: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s 2013-04-03 10:43:06.692985 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos 2013-04-03 10:43:06.693004 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 2013-04-03 10:43:06.693025 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:06.693068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos 2013-04-03 10:43:06.693085 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 2013-04-03 10:43:06.693094 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:06.693106 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:43:06.693130 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:06.693141 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:43:06.693151 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 logging 2013-04-03 10:43:06.692925 mon.0 10.252.24.25:6789/0 149 : [INF] pgmap v29083: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s 2013-04-03 10:43:06.718642 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos 2013-04-03 10:43:06.718659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 2013-04-03 10:43:06.718674 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:07.592508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140430 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:43:07.692577 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394078 encode_pending v394079 2013-04-03 10:43:07.721189 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos 2013-04-03 10:43:07.721214 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394078 2013-04-03 10:43:07.721229 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos latest full 393798 2013-04-03 10:43:07.721248 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394079 update_from_paxos applying incremental log 394079 2013-04-03 10:43:06.692925 mon.0 10.252.24.25:6789/0 149 : [INF] pgmap v29083: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s 2013-04-03 10:43:07.721295 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 check_subs 2013-04-03 10:43:07.721326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 create_pending v 394080 2013-04-03 10:43:07.721349 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394079 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:43:07.721413 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos 2013-04-03 10:43:07.721429 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 2013-04-03 10:43:07.721447 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:07.733191 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos 2013-04-03 10:43:07.733207 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 2013-04-03 10:43:07.733224 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:08.364336 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29083 check_down_pgs 2013-04-03 10:43:08.364372 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29083 v29083: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s 2013-04-03 10:43:08.364400 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:43:08.364420 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:43:08.364456 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:43:08.364486 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos 2013-04-03 10:43:08.364502 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 2013-04-03 10:43:08.364511 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394079 log 2013-04-03 10:43:08.364527 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:08.364544 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:43:08.938830 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:08.938840 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae340 from 10.252.24.88:6789/0 2013-04-03 10:43:08.938892 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 145 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:43:09.391086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21801 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:43:10.722084 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos 2013-04-03 10:43:10.722104 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 2013-04-03 10:43:10.722122 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:10.937173 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33103/0) 2013-04-03 10:43:11.592660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140431 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:43:12.939021 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:12.939030 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae600 from 10.252.24.88:6789/0 2013-04-03 10:43:12.939084 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 146 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:43:13.258637 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:13.258644 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:43:13.258680 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 preprocess_query pg_stats(1 pgs tid 26 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:43:13.258704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 prepare_update pg_stats(1 pgs tid 26 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:43:13.258710 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 prepare_pg_stats pg_stats(1 pgs tid 26 v 605) v1 from osd.2 2013-04-03 10:43:13.258716 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:43:13.308898 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29083 encode_pending v 29084 2013-04-03 10:43:13.339853 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29083 update_from_paxos applying incremental 29084 2013-04-03 10:43:13.339869 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 v29084: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:43:13.339888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 send_pg_creates to 0 pgs 2013-04-03 10:43:13.339892 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 update_logger 2013-04-03 10:43:13.339902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 create_pending v 29085 2013-04-03 10:43:13.339910 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29084 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:43:13.339926 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 check_osd_map already seen 605 >= 605 2013-04-03 10:43:13.339931 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 update_logger 2013-04-03 10:43:13.339941 7fcdffcdb700 0 log [INF] : pgmap v29084: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:43:13.339998 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos 2013-04-03 10:43:13.340017 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 2013-04-03 10:43:13.340036 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:13.340081 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos 2013-04-03 10:43:13.340096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 2013-04-03 10:43:13.340104 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:13.340114 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:43:13.340136 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:13.340148 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:43:13.340157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 logging 2013-04-03 10:43:13.339942 mon.0 10.252.24.25:6789/0 150 : [INF] pgmap v29084: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:43:13.364655 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29084 check_down_pgs 2013-04-03 10:43:13.364684 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29084 v29084: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:43:13.364710 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:43:13.364730 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:43:13.364766 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:43:13.364797 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos 2013-04-03 10:43:13.364816 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 2013-04-03 10:43:13.364826 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394079 log 2013-04-03 10:43:13.364841 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:13.364858 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:43:13.365553 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos 2013-04-03 10:43:13.365571 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 2013-04-03 10:43:13.365588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:13.391190 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21802 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:43:14.339946 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394079 encode_pending v394080 2013-04-03 10:43:14.365957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos 2013-04-03 10:43:14.365985 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394079 2013-04-03 10:43:14.366002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos latest full 393798 2013-04-03 10:43:14.366021 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394080 update_from_paxos applying incremental log 394080 2013-04-03 10:43:13.339942 mon.0 10.252.24.25:6789/0 150 : [INF] pgmap v29084: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:43:14.366066 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 check_subs 2013-04-03 10:43:14.366098 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 create_pending v 394081 2013-04-03 10:43:14.366120 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394080 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:43:14.366186 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos 2013-04-03 10:43:14.366203 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 2013-04-03 10:43:14.366221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:14.377922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos 2013-04-03 10:43:14.377938 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 2013-04-03 10:43:14.377954 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:15.592790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140432 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:43:16.939092 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:16.939102 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2145dc0 from 10.252.24.88:6789/0 2013-04-03 10:43:16.939154 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 147 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:43:17.366784 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos 2013-04-03 10:43:17.366803 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 2013-04-03 10:43:17.366820 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:17.391255 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21803 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:43:18.364995 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29084 v29084: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:43:18.365038 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:43:18.365060 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:43:18.365104 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:43:18.365135 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos 2013-04-03 10:43:18.365154 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 2013-04-03 10:43:18.365163 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394080 log 2013-04-03 10:43:18.365180 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:18.365199 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:43:19.592897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140433 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:43:20.366946 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos 2013-04-03 10:43:20.366966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 2013-04-03 10:43:20.366984 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:20.939176 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:20.939186 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x21452c0 from 10.252.24.88:6789/0 2013-04-03 10:43:20.939239 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 148 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:43:21.391363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21804 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:43:21.615072 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:21.615082 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:43:21.615134 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 preprocess_query pg_stats(1 pgs tid 46 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:43:21.615158 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 prepare_update pg_stats(1 pgs tid 46 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:43:21.615164 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 prepare_pg_stats pg_stats(1 pgs tid 46 v 605) v1 from osd.1 2013-04-03 10:43:21.615170 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:43:21.665376 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29084 encode_pending v 29085 2013-04-03 10:43:21.695962 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29084 update_from_paxos applying incremental 29085 2013-04-03 10:43:21.695981 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 v29085: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:43:21.696000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 send_pg_creates to 0 pgs 2013-04-03 10:43:21.696003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 update_logger 2013-04-03 10:43:21.696012 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 create_pending v 29086 2013-04-03 10:43:21.696021 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29085 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:43:21.696038 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 check_osd_map already seen 605 >= 605 2013-04-03 10:43:21.696040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 update_logger 2013-04-03 10:43:21.696050 7fcdffcdb700 0 log [INF] : pgmap v29085: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:43:21.696119 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos 2013-04-03 10:43:21.696138 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 2013-04-03 10:43:21.696157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:21.696200 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos 2013-04-03 10:43:21.696215 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 2013-04-03 10:43:21.696224 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:21.696236 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:43:21.696260 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:21.696271 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:43:21.696281 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 logging 2013-04-03 10:43:21.696057 mon.0 10.252.24.25:6789/0 151 : [INF] pgmap v29085: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:43:21.708686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos 2013-04-03 10:43:21.708704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 2013-04-03 10:43:21.708719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:21.780759 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:21.780766 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:43:21.780795 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 preprocess_query pg_stats(1 pgs tid 30455 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:43:21.780816 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 prepare_update pg_stats(1 pgs tid 30455 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:43:21.780821 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 prepare_pg_stats pg_stats(1 pgs tid 30455 v 605) v1 from osd.3 2013-04-03 10:43:21.780827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:43:22.695161 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394080 encode_pending v394081 2013-04-03 10:43:22.696404 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29085 encode_pending v 29086 2013-04-03 10:43:22.725740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos 2013-04-03 10:43:22.725766 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos version 394081 summary v 394080 2013-04-03 10:43:22.725781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos latest full 393798 2013-04-03 10:43:22.725799 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394081 update_from_paxos applying incremental log 394081 2013-04-03 10:43:21.696057 mon.0 10.252.24.25:6789/0 151 : [INF] pgmap v29085: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:43:22.725856 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 check_subs 2013-04-03 10:43:22.725889 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 create_pending v 394082 2013-04-03 10:43:22.725911 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394081 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:43:22.910699 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29085 update_from_paxos applying incremental 29086 2013-04-03 10:43:22.910718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 v29086: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:43:22.910736 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 send_pg_creates to 0 pgs 2013-04-03 10:43:22.910740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 update_logger 2013-04-03 10:43:22.910750 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 create_pending v 29087 2013-04-03 10:43:22.910759 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29086 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:43:22.910775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 check_osd_map already seen 605 >= 605 2013-04-03 10:43:22.910778 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 update_logger 2013-04-03 10:43:22.910788 7fcdffcdb700 0 log [INF] : pgmap v29086: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:43:22.910855 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos 2013-04-03 10:43:22.910872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos version 394081 summary v 394081 2013-04-03 10:43:22.910891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:22.910933 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos 2013-04-03 10:43:22.910947 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos version 394081 summary v 394081 2013-04-03 10:43:22.910956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:22.910968 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:43:22.910988 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:22.911000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:43:22.911011 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 logging 2013-04-03 10:43:22.910789 mon.0 10.252.24.25:6789/0 152 : [INF] pgmap v29086: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:43:22.997936 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos 2013-04-03 10:43:22.997952 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos version 394081 summary v 394081 2013-04-03 10:43:22.997967 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:23.365321 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29086 check_down_pgs 2013-04-03 10:43:23.365351 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29086 v29086: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:43:23.365377 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:43:23.365398 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:43:23.365435 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:43:23.365465 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos 2013-04-03 10:43:23.365482 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos version 394081 summary v 394081 2013-04-03 10:43:23.365491 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394081 log 2013-04-03 10:43:23.365511 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:23.365528 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:43:23.592999 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140434 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:43:23.910504 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394081 encode_pending v394082 2013-04-03 10:43:23.943403 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos 2013-04-03 10:43:23.943428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394081 2013-04-03 10:43:23.943443 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos latest full 393798 2013-04-03 10:43:23.943461 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394082 update_from_paxos applying incremental log 394082 2013-04-03 10:43:22.910789 mon.0 10.252.24.25:6789/0 152 : [INF] pgmap v29086: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:43:23.943507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 check_subs 2013-04-03 10:43:23.943538 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 create_pending v 394083 2013-04-03 10:43:23.943560 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394082 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:43:23.943624 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos 2013-04-03 10:43:23.943640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394082 2013-04-03 10:43:23.943658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:23.969124 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos 2013-04-03 10:43:23.969141 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394082 2013-04-03 10:43:23.969157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:24.939322 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:24.939334 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e3b00 from 10.252.24.88:6789/0 2013-04-03 10:43:24.939386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 149 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:43:25.391402 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21805 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:43:26.944242 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos 2013-04-03 10:43:26.944262 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394082 2013-04-03 10:43:26.944279 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:27.593117 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140435 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:43:28.260783 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:28.260792 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:43:28.260842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 preprocess_query pg_stats(1 pgs tid 27 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:43:28.260867 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 prepare_update pg_stats(1 pgs tid 27 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:43:28.260873 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 prepare_pg_stats pg_stats(1 pgs tid 27 v 605) v1 from osd.2 2013-04-03 10:43:28.260878 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:43:28.311063 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29086 encode_pending v 29087 2013-04-03 10:43:28.365642 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:43:28.365683 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:43:28.365906 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos 2013-04-03 10:43:28.365929 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394082 2013-04-03 10:43:28.365938 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394082 log 2013-04-03 10:43:28.365956 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:28.365975 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:43:28.508000 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29086 update_from_paxos applying incremental 29087 2013-04-03 10:43:28.508022 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29087 v29087: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:43:28.508040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29087 send_pg_creates to 0 pgs 2013-04-03 10:43:28.508044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29087 update_logger 2013-04-03 10:43:28.508055 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29087 create_pending v 29088 2013-04-03 10:43:28.508064 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29087 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:43:28.508082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29087 check_osd_map already seen 605 >= 605 2013-04-03 10:43:28.508085 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29087 update_logger 2013-04-03 10:43:28.508102 7fcdffcdb700 0 log [INF] : pgmap v29087: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:43:28.508167 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos 2013-04-03 10:43:28.508186 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394082 2013-04-03 10:43:28.508204 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:28.508247 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos 2013-04-03 10:43:28.508262 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394082 2013-04-03 10:43:28.508271 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:28.508282 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:43:28.508307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:28.508320 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:43:28.508330 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 logging 2013-04-03 10:43:28.508104 mon.0 10.252.24.25:6789/0 153 : [INF] pgmap v29087: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:43:28.519828 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos 2013-04-03 10:43:28.519844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394082 2013-04-03 10:43:28.519862 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:28.759379 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:43:28.759386 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:43:28.759395 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 2013-04-03 10:43:28.939441 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:28.939450 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20e4600 from 10.252.24.88:6789/0 2013-04-03 10:43:28.939503 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 150 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:43:29.391537 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21806 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:43:29.507725 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394082 encode_pending v394083 2013-04-03 10:43:29.537978 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos 2013-04-03 10:43:29.538011 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394082 2013-04-03 10:43:29.538029 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos latest full 393798 2013-04-03 10:43:29.538049 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos applying incremental log 394083 2013-04-03 10:43:28.508104 mon.0 10.252.24.25:6789/0 153 : [INF] pgmap v29087: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:43:29.538095 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 check_subs 2013-04-03 10:43:29.538126 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 create_pending v 394084 2013-04-03 10:43:29.538149 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:43:29.538215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos 2013-04-03 10:43:29.538231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 2013-04-03 10:43:29.538250 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:29.563763 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos 2013-04-03 10:43:29.563780 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 2013-04-03 10:43:29.563798 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:30.969227 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003c80 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33110/0) 2013-04-03 10:43:31.593278 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140436 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:43:32.538909 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos 2013-04-03 10:43:32.538930 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 2013-04-03 10:43:32.538948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:32.939696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:32.939707 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20e3dc0 from 10.252.24.88:6789/0 2013-04-03 10:43:32.939753 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 151 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:43:33.366097 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 check_down_pgs 2013-04-03 10:43:33.366131 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 v29087: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s 2013-04-03 10:43:33.366157 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:43:33.366181 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:43:33.366218 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:43:33.366250 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos 2013-04-03 10:43:33.366270 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 2013-04-03 10:43:33.366279 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 log 2013-04-03 10:43:33.366296 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:33.366316 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:43:33.391653 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21807 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:43:35.507688 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:35.507700 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 2013-04-03 10:43:35.507754 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 preprocess_query pg_stats(0 pgs tid 62 v 605) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:43:35.507779 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 prepare_update pg_stats(0 pgs tid 62 v 605) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:43:35.507786 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 prepare_pg_stats pg_stats(0 pgs tid 62 v 605) v1 from osd.4 2013-04-03 10:43:35.507792 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 got osd.4 osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[])) 2013-04-03 10:43:35.538839 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos 2013-04-03 10:43:35.538860 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 2013-04-03 10:43:35.538878 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:35.557972 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 encode_pending v 29088 2013-04-03 10:43:35.585960 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 update_from_paxos applying incremental 29088 2013-04-03 10:43:35.585972 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 v29088: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 955B/s wr, 0op/s 2013-04-03 10:43:35.585990 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 send_pg_creates to 0 pgs 2013-04-03 10:43:35.585993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 update_logger 2013-04-03 10:43:35.586003 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 create_pending v 29089 2013-04-03 10:43:35.586015 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 _updated_stats for osd.4 10.252.24.32:6800/13473 2013-04-03 10:43:35.586032 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 check_osd_map already seen 605 >= 605 2013-04-03 10:43:35.586038 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 update_logger 2013-04-03 10:43:35.586048 7fcdffcdb700 0 log [INF] : pgmap v29088: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 955B/s wr, 0op/s 2013-04-03 10:43:35.586106 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos 2013-04-03 10:43:35.586125 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 2013-04-03 10:43:35.586144 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:35.586186 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos 2013-04-03 10:43:35.586201 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 2013-04-03 10:43:35.586211 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:35.586223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:43:35.586248 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:35.586262 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:43:35.586274 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 logging 2013-04-03 10:43:35.586049 mon.0 10.252.24.25:6789/0 154 : [INF] pgmap v29088: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 955B/s wr, 0op/s 2013-04-03 10:43:35.593333 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140437 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:43:35.598567 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos 2013-04-03 10:43:35.598584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 2013-04-03 10:43:35.598600 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:36.585077 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 encode_pending v394084 2013-04-03 10:43:36.615681 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos 2013-04-03 10:43:36.615709 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos version 394084 summary v 394083 2013-04-03 10:43:36.615725 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos latest full 393798 2013-04-03 10:43:36.615744 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos applying incremental log 394084 2013-04-03 10:43:35.586049 mon.0 10.252.24.25:6789/0 154 : [INF] pgmap v29088: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 955B/s wr, 0op/s 2013-04-03 10:43:36.615792 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 check_subs 2013-04-03 10:43:36.615824 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 create_pending v 394085 2013-04-03 10:43:36.615847 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:43:36.615912 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos 2013-04-03 10:43:36.616139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos version 394084 summary v 394084 2013-04-03 10:43:36.616159 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:36.627791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos 2013-04-03 10:43:36.627808 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos version 394084 summary v 394084 2013-04-03 10:43:36.627824 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:36.628008 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:36.628016 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:43:36.628046 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 preprocess_query pg_stats(1 pgs tid 47 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:43:36.628068 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 prepare_update pg_stats(1 pgs tid 47 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:43:36.628074 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 prepare_pg_stats pg_stats(1 pgs tid 47 v 605) v1 from osd.1 2013-04-03 10:43:36.628081 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:43:36.843003 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:36.843016 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:43:36.843065 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 preprocess_query pg_stats(0 pgs tid 411 v 605) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:43:36.843090 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 prepare_update pg_stats(0 pgs tid 411 v 605) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:43:36.843103 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 prepare_pg_stats pg_stats(0 pgs tid 411 v 605) v1 from osd.0 2013-04-03 10:43:36.843110 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 got osd.0 osd_stat(1141 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1140 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:43:36.939709 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:36.939717 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83580 from 10.252.24.88:6789/0 2013-04-03 10:43:36.939751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 152 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:43:37.391875 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21808 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:43:37.615538 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 encode_pending v 29089 2013-04-03 10:43:37.736548 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 update_from_paxos applying incremental 29089 2013-04-03 10:43:37.736570 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 v29089: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 440B/s wr, 0op/s 2013-04-03 10:43:37.736591 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 send_pg_creates to 0 pgs 2013-04-03 10:43:37.736595 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 update_logger 2013-04-03 10:43:37.736606 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 create_pending v 29090 2013-04-03 10:43:37.736616 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:43:37.736635 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:43:37.736669 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 check_osd_map already seen 605 >= 605 2013-04-03 10:43:37.736673 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 update_logger 2013-04-03 10:43:37.736684 7fcdffcdb700 0 log [INF] : pgmap v29089: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 440B/s wr, 0op/s 2013-04-03 10:43:37.736748 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos 2013-04-03 10:43:37.736768 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos version 394084 summary v 394084 2013-04-03 10:43:37.736787 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:37.736831 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos 2013-04-03 10:43:37.736846 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos version 394084 summary v 394084 2013-04-03 10:43:37.736856 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:37.736868 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:43:37.736894 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:37.736909 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:43:37.736921 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 logging 2013-04-03 10:43:37.736685 mon.0 10.252.24.25:6789/0 155 : [INF] pgmap v29089: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 440B/s wr, 0op/s 2013-04-03 10:43:37.762342 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos 2013-04-03 10:43:37.762358 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos version 394084 summary v 394084 2013-04-03 10:43:37.762373 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:38.366465 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 check_down_pgs 2013-04-03 10:43:38.366492 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 v29089: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 440B/s wr, 0op/s 2013-04-03 10:43:38.366520 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:43:38.366545 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:43:38.366582 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:43:38.366614 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos 2013-04-03 10:43:38.366633 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos version 394084 summary v 394084 2013-04-03 10:43:38.366643 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 log 2013-04-03 10:43:38.366660 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:38.366679 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:43:38.736318 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 encode_pending v394085 2013-04-03 10:43:38.762627 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos 2013-04-03 10:43:38.762655 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394084 2013-04-03 10:43:38.762670 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos latest full 393798 2013-04-03 10:43:38.763169 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos applying incremental log 394085 2013-04-03 10:43:37.736685 mon.0 10.252.24.25:6789/0 155 : [INF] pgmap v29089: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 440B/s wr, 0op/s 2013-04-03 10:43:38.763226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 check_subs 2013-04-03 10:43:38.763259 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 create_pending v 394086 2013-04-03 10:43:38.763282 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:43:38.763347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos 2013-04-03 10:43:38.763364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394085 2013-04-03 10:43:38.763385 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:38.774680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos 2013-04-03 10:43:38.774698 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394085 2013-04-03 10:43:38.774715 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:39.593491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140438 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:43:40.939854 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:40.939865 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc4580 from 10.252.24.88:6789/0 2013-04-03 10:43:40.939919 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 153 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:43:40.985200 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003a00 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33111/0) 2013-04-03 10:43:41.391988 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21809 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:43:41.763982 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos 2013-04-03 10:43:41.764003 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394085 2013-04-03 10:43:41.764021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:43.262663 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:43.262674 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:43:43.262725 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 preprocess_query pg_stats(1 pgs tid 28 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:43:43.262750 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 prepare_update pg_stats(1 pgs tid 28 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:43:43.262757 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 prepare_pg_stats pg_stats(1 pgs tid 28 v 605) v1 from osd.2 2013-04-03 10:43:43.262764 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:43:43.312951 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 encode_pending v 29090 2013-04-03 10:43:43.341218 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 update_from_paxos applying incremental 29090 2013-04-03 10:43:43.341236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 v29090: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s 2013-04-03 10:43:43.341257 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 send_pg_creates to 0 pgs 2013-04-03 10:43:43.341261 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 update_logger 2013-04-03 10:43:43.341271 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 create_pending v 29091 2013-04-03 10:43:43.341281 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:43:43.341297 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 check_osd_map already seen 605 >= 605 2013-04-03 10:43:43.341300 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 update_logger 2013-04-03 10:43:43.341311 7fcdffcdb700 0 log [INF] : pgmap v29090: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s 2013-04-03 10:43:43.341370 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos 2013-04-03 10:43:43.341390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394085 2013-04-03 10:43:43.341612 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:43.341658 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos 2013-04-03 10:43:43.341673 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394085 2013-04-03 10:43:43.341685 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:43.341697 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:43:43.341722 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:43.341735 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:43:43.341745 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 logging 2013-04-03 10:43:43.341312 mon.0 10.252.24.25:6789/0 156 : [INF] pgmap v29090: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s 2013-04-03 10:43:43.366407 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos 2013-04-03 10:43:43.366424 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394085 2013-04-03 10:43:43.366441 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:43.366790 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 check_down_pgs 2013-04-03 10:43:43.366821 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 v29090: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s 2013-04-03 10:43:43.366852 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:43:43.366874 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:43:43.366912 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:43:43.366945 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos 2013-04-03 10:43:43.366962 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394085 2013-04-03 10:43:43.366972 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 log 2013-04-03 10:43:43.366988 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:43.367006 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:43:43.593641 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140439 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:43:44.340421 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 encode_pending v394086 2013-04-03 10:43:44.366846 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos 2013-04-03 10:43:44.366874 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394085 2013-04-03 10:43:44.366890 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos latest full 393798 2013-04-03 10:43:44.366910 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos applying incremental log 394086 2013-04-03 10:43:43.341312 mon.0 10.252.24.25:6789/0 156 : [INF] pgmap v29090: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s 2013-04-03 10:43:44.366957 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 check_subs 2013-04-03 10:43:44.366989 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 create_pending v 394087 2013-04-03 10:43:44.367012 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:43:44.367078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos 2013-04-03 10:43:44.367094 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 2013-04-03 10:43:44.367112 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:44.380189 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos 2013-04-03 10:43:44.380206 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 2013-04-03 10:43:44.380223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:44.940037 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:44.940048 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83000 from 10.252.24.88:6789/0 2013-04-03 10:43:44.940101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 154 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:43:45.392104 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21810 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:43:47.367816 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos 2013-04-03 10:43:47.367837 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 2013-04-03 10:43:47.367855 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:47.419461 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 2013-04-03 10:43:47.419472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 2013-04-03 10:43:47.593752 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140440 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:43:48.367146 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 v29090: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s 2013-04-03 10:43:48.367193 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:43:48.367216 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:43:48.367260 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:43:48.367291 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos 2013-04-03 10:43:48.367311 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 2013-04-03 10:43:48.367321 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 log 2013-04-03 10:43:48.367338 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:48.367357 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:43:48.940166 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:48.940177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc5080 from 10.252.24.88:6789/0 2013-04-03 10:43:48.940231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 155 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:43:49.392254 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21811 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:43:50.367830 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos 2013-04-03 10:43:50.367851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 2013-04-03 10:43:50.367870 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:51.593888 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140441 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:43:51.618211 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:51.618220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:43:51.618254 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 preprocess_query pg_stats(1 pgs tid 48 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:43:51.618279 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 prepare_update pg_stats(1 pgs tid 48 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:43:51.618285 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 prepare_pg_stats pg_stats(1 pgs tid 48 v 605) v1 from osd.1 2013-04-03 10:43:51.618292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:43:51.668480 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 encode_pending v 29091 2013-04-03 10:43:51.705115 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 update_from_paxos applying incremental 29091 2013-04-03 10:43:51.705132 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 v29091: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1165B/s wr, 0op/s 2013-04-03 10:43:51.705153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 send_pg_creates to 0 pgs 2013-04-03 10:43:51.705157 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 update_logger 2013-04-03 10:43:51.705168 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 create_pending v 29092 2013-04-03 10:43:51.705177 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:43:51.705193 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 check_osd_map already seen 605 >= 605 2013-04-03 10:43:51.705197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 update_logger 2013-04-03 10:43:51.705208 7fcdffcdb700 0 log [INF] : pgmap v29091: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1165B/s wr, 0op/s 2013-04-03 10:43:51.705265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos 2013-04-03 10:43:51.705284 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 2013-04-03 10:43:51.705304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:51.705349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos 2013-04-03 10:43:51.705364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 2013-04-03 10:43:51.705374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:51.705386 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:43:51.705410 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:51.705423 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:43:51.705433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 logging 2013-04-03 10:43:51.705209 mon.0 10.252.24.25:6789/0 157 : [INF] pgmap v29091: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1165B/s wr, 0op/s 2013-04-03 10:43:51.731017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos 2013-04-03 10:43:51.731038 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 2013-04-03 10:43:51.731054 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:51.784059 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:51.784066 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:43:51.784095 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 preprocess_query pg_stats(1 pgs tid 30456 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:43:51.784116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 prepare_update pg_stats(1 pgs tid 30456 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:43:51.784122 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 prepare_pg_stats pg_stats(1 pgs tid 30456 v 605) v1 from osd.3 2013-04-03 10:43:51.784128 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:43:52.704890 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 encode_pending v394087 2013-04-03 10:43:52.706551 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 encode_pending v 29092 2013-04-03 10:43:52.731395 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos 2013-04-03 10:43:52.731422 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos version 394087 summary v 394086 2013-04-03 10:43:52.731439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos latest full 393798 2013-04-03 10:43:52.731458 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos applying incremental log 394087 2013-04-03 10:43:51.705209 mon.0 10.252.24.25:6789/0 157 : [INF] pgmap v29091: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1165B/s wr, 0op/s 2013-04-03 10:43:52.731505 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 check_subs 2013-04-03 10:43:52.731538 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 create_pending v 394088 2013-04-03 10:43:52.731562 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:43:52.758464 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 update_from_paxos applying incremental 29092 2013-04-03 10:43:52.758485 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 v29092: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:43:52.758505 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 send_pg_creates to 0 pgs 2013-04-03 10:43:52.758509 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 update_logger 2013-04-03 10:43:52.758519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 create_pending v 29093 2013-04-03 10:43:52.758918 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:43:52.758940 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 check_osd_map already seen 605 >= 605 2013-04-03 10:43:52.758943 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 update_logger 2013-04-03 10:43:52.758955 7fcdffcdb700 0 log [INF] : pgmap v29092: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:43:52.759017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos 2013-04-03 10:43:52.759035 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos version 394087 summary v 394087 2013-04-03 10:43:52.759054 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:52.759098 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos 2013-04-03 10:43:52.759116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos version 394087 summary v 394087 2013-04-03 10:43:52.759125 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:52.759138 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:43:52.759159 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:52.759172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:43:52.759184 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 logging 2013-04-03 10:43:52.758956 mon.0 10.252.24.25:6789/0 158 : [INF] pgmap v29092: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:43:52.769359 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos 2013-04-03 10:43:52.769377 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos version 394087 summary v 394087 2013-04-03 10:43:52.769394 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:52.940227 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:52.940236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2996600 from 10.252.24.88:6789/0 2013-04-03 10:43:52.940268 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 156 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:43:53.332508 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_start_round curr 4 2013-04-03 10:43:53.332531 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_start_round new 5 2013-04-03 10:43:53.332538 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck 2013-04-03 10:43:53.332542 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck start timecheck epoch 352 round 5 2013-04-03 10:43:53.332553 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck send time_check( ping e 352 r 5 ) v1 to mon.2 10.252.24.88:6789/0 2013-04-03 10:43:53.332582 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_start_round setting up next event 2013-04-03 10:43:53.333327 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck time_check( pong e 352 r 5 ts 2013-04-03 10:43:53.332381 ) v1 2013-04-03 10:43:53.333333 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck_leader time_check( pong e 352 r 5 ts 2013-04-03 10:43:53.332381 ) v1 2013-04-03 10:43:53.333340 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck_leader from mon.2 10.252.24.88:6789/0 ts 2013-04-03 10:43:53.332381 delta -0.000956059 skew_bound -0.000170059 latency 0.000786 2013-04-03 10:43:53.333357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck_leader got pongs from everybody (2 total) 2013-04-03 10:43:53.333360 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_finish_round curr 5 2013-04-03 10:43:53.333361 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_report 2013-04-03 10:43:53.333366 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_report send report to mon.2 10.252.24.88:6789/0 2013-04-03 10:43:53.367490 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 check_down_pgs 2013-04-03 10:43:53.367520 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 v29092: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:43:53.367550 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:43:53.367574 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:43:53.367611 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:43:53.367642 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos 2013-04-03 10:43:53.367663 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos version 394087 summary v 394087 2013-04-03 10:43:53.367673 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 log 2013-04-03 10:43:53.367693 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:53.367713 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:43:53.392315 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21812 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:43:53.757058 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 encode_pending v394088 2013-04-03 10:43:53.782611 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos 2013-04-03 10:43:53.782639 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394087 2013-04-03 10:43:53.782656 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos latest full 393798 2013-04-03 10:43:53.782675 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos applying incremental log 394088 2013-04-03 10:43:52.758956 mon.0 10.252.24.25:6789/0 158 : [INF] pgmap v29092: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s 2013-04-03 10:43:53.782722 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 check_subs 2013-04-03 10:43:53.782755 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 create_pending v 394089 2013-04-03 10:43:53.782778 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:43:53.782847 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos 2013-04-03 10:43:53.782864 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394088 2013-04-03 10:43:53.782882 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:53.794484 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos 2013-04-03 10:43:53.794501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394088 2013-04-03 10:43:53.794517 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:55.484970 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 2013-04-03 10:43:55.484981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 2013-04-03 10:43:55.593965 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140442 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:43:56.783447 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos 2013-04-03 10:43:56.783471 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394088 2013-04-03 10:43:56.783489 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:56.940338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:56.940347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2995b00 from 10.252.24.88:6789/0 2013-04-03 10:43:56.940380 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 157 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:43:57.392445 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21813 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:43:58.264545 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:43:58.264556 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 2013-04-03 10:43:58.264607 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 preprocess_query pg_stats(1 pgs tid 29 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:43:58.264632 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 prepare_update pg_stats(1 pgs tid 29 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:43:58.264639 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 prepare_pg_stats pg_stats(1 pgs tid 29 v 605) v1 from osd.2 2013-04-03 10:43:58.264645 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:43:58.314833 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 encode_pending v 29093 2013-04-03 10:43:58.341521 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 update_from_paxos applying incremental 29093 2013-04-03 10:43:58.341536 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 v29093: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2619B/s wr, 0op/s 2013-04-03 10:43:58.341557 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 send_pg_creates to 0 pgs 2013-04-03 10:43:58.341561 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 update_logger 2013-04-03 10:43:58.341572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 create_pending v 29094 2013-04-03 10:43:58.341581 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:43:58.341598 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 check_osd_map already seen 605 >= 605 2013-04-03 10:43:58.341600 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 update_logger 2013-04-03 10:43:58.341611 7fcdffcdb700 0 log [INF] : pgmap v29093: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2619B/s wr, 0op/s 2013-04-03 10:43:58.341668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos 2013-04-03 10:43:58.341688 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394088 2013-04-03 10:43:58.341707 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:58.341754 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos 2013-04-03 10:43:58.341770 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394088 2013-04-03 10:43:58.341779 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:58.341791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:43:58.341827 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:43:58.341843 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:43:58.341855 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 logging 2013-04-03 10:43:58.341612 mon.0 10.252.24.25:6789/0 159 : [INF] pgmap v29093: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2619B/s wr, 0op/s 2013-04-03 10:43:58.354865 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos 2013-04-03 10:43:58.354882 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394088 2013-04-03 10:43:58.354897 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:58.367825 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 check_down_pgs 2013-04-03 10:43:58.367855 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 v29093: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2619B/s wr, 0op/s 2013-04-03 10:43:58.367881 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:43:58.367903 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:43:58.367941 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:43:58.367972 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos 2013-04-03 10:43:58.367991 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394088 2013-04-03 10:43:58.368000 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 log 2013-04-03 10:43:58.368017 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:43:58.368034 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:43:58.759465 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) 2013-04-03 10:43:58.759504 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader 2013-04-03 10:43:59.341346 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394088 encode_pending v394089 2013-04-03 10:43:59.373035 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos 2013-04-03 10:43:59.373062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394088 2013-04-03 10:43:59.373076 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos latest full 393798 2013-04-03 10:43:59.373095 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394089 update_from_paxos applying incremental log 394089 2013-04-03 10:43:58.341612 mon.0 10.252.24.25:6789/0 159 : [INF] pgmap v29093: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2619B/s wr, 0op/s 2013-04-03 10:43:59.373355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 check_subs 2013-04-03 10:43:59.373388 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 create_pending v 394090 2013-04-03 10:43:59.373410 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394089 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:43:59.373475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos 2013-04-03 10:43:59.373491 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 2013-04-03 10:43:59.373509 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:59.385031 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos 2013-04-03 10:43:59.385047 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 2013-04-03 10:43:59.385080 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:43:59.594119 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140443 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:44:00.940497 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:00.940506 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2995000 from 10.252.24.88:6789/0 2013-04-03 10:44:00.940561 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 158 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:44:01.017205 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33115/0) 2013-04-03 10:44:01.392526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21814 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:44:02.373924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos 2013-04-03 10:44:02.373943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 2013-04-03 10:44:02.373960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:03.368175 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29093 v29093: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2619B/s wr, 0op/s 2013-04-03 10:44:03.368221 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:44:03.368243 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:44:03.368283 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:44:03.368315 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos 2013-04-03 10:44:03.368333 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 2013-04-03 10:44:03.368343 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394089 log 2013-04-03 10:44:03.368360 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:03.368381 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:44:03.594255 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140444 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:44:04.940557 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:04.940566 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a84600 from 10.252.24.88:6789/0 2013-04-03 10:44:04.940621 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 159 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:44:05.373998 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos 2013-04-03 10:44:05.374020 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 2013-04-03 10:44:05.374038 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:05.392618 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21815 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:44:06.619984 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:06.619994 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:44:06.620045 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29093 preprocess_query pg_stats(1 pgs tid 49 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:44:06.620069 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29093 prepare_update pg_stats(1 pgs tid 49 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:44:06.620075 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29093 prepare_pg_stats pg_stats(1 pgs tid 49 v 605) v1 from osd.1 2013-04-03 10:44:06.620081 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29093 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:44:06.670268 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29093 encode_pending v 29094 2013-04-03 10:44:06.807832 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29093 update_from_paxos applying incremental 29094 2013-04-03 10:44:06.807848 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 v29094: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1466B/s wr, 0op/s 2013-04-03 10:44:06.807866 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 send_pg_creates to 0 pgs 2013-04-03 10:44:06.807870 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 update_logger 2013-04-03 10:44:06.807882 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 create_pending v 29095 2013-04-03 10:44:06.807891 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29094 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:44:06.807907 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 check_osd_map already seen 605 >= 605 2013-04-03 10:44:06.807910 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 update_logger 2013-04-03 10:44:06.807920 7fcdffcdb700 0 log [INF] : pgmap v29094: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1466B/s wr, 0op/s 2013-04-03 10:44:06.807978 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos 2013-04-03 10:44:06.807997 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 2013-04-03 10:44:06.808016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:06.808058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos 2013-04-03 10:44:06.808073 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 2013-04-03 10:44:06.808082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:06.808092 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:44:06.808115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:06.808130 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:44:06.808140 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 logging 2013-04-03 10:44:06.807921 mon.0 10.252.24.25:6789/0 160 : [INF] pgmap v29094: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1466B/s wr, 0op/s 2013-04-03 10:44:06.819822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos 2013-04-03 10:44:06.819839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 2013-04-03 10:44:06.819853 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:07.594353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140445 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:44:07.807566 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394089 encode_pending v394090 2013-04-03 10:44:07.838590 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos 2013-04-03 10:44:07.838617 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394089 2013-04-03 10:44:07.838632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos latest full 393798 2013-04-03 10:44:07.838652 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394090 update_from_paxos applying incremental log 394090 2013-04-03 10:44:06.807921 mon.0 10.252.24.25:6789/0 160 : [INF] pgmap v29094: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1466B/s wr, 0op/s 2013-04-03 10:44:07.838699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 check_subs 2013-04-03 10:44:07.838731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 create_pending v 394091 2013-04-03 10:44:07.838754 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394090 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:44:07.838819 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos 2013-04-03 10:44:07.838835 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 2013-04-03 10:44:07.838853 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:07.863804 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos 2013-04-03 10:44:07.863820 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 2013-04-03 10:44:07.863837 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:08.368529 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29094 check_down_pgs 2013-04-03 10:44:08.368556 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29094 v29094: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1466B/s wr, 0op/s 2013-04-03 10:44:08.368584 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:44:08.368611 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:44:08.368647 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:44:08.368680 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos 2013-04-03 10:44:08.368698 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 2013-04-03 10:44:08.368707 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394090 log 2013-04-03 10:44:08.368724 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:08.368743 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:44:08.940844 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:08.940854 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a51840 from 10.252.24.88:6789/0 2013-04-03 10:44:08.940908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 160 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:44:09.392827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21816 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:44:10.839432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos 2013-04-03 10:44:10.839452 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 2013-04-03 10:44:10.839469 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:11.594507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140446 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:44:12.940890 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:12.940899 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2995dc0 from 10.252.24.88:6789/0 2013-04-03 10:44:12.940952 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 161 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:44:13.266559 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:13.266569 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 2013-04-03 10:44:13.266620 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 preprocess_query pg_stats(1 pgs tid 30 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:44:13.266645 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 prepare_update pg_stats(1 pgs tid 30 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:44:13.266650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 prepare_pg_stats pg_stats(1 pgs tid 30 v 605) v1 from osd.2 2013-04-03 10:44:13.266657 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:44:13.316847 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29094 encode_pending v 29095 2013-04-03 10:44:13.345403 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29094 update_from_paxos applying incremental 29095 2013-04-03 10:44:13.345418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 v29095: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:44:13.345437 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 send_pg_creates to 0 pgs 2013-04-03 10:44:13.345440 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 update_logger 2013-04-03 10:44:13.345449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 create_pending v 29096 2013-04-03 10:44:13.345458 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29095 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:44:13.345477 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 check_osd_map already seen 605 >= 605 2013-04-03 10:44:13.345480 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 update_logger 2013-04-03 10:44:13.345491 7fcdffcdb700 0 log [INF] : pgmap v29095: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:44:13.345686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos 2013-04-03 10:44:13.345709 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 2013-04-03 10:44:13.345729 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:13.345775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos 2013-04-03 10:44:13.345790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 2013-04-03 10:44:13.345798 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:13.345808 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:44:13.345843 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:13.345857 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:44:13.345871 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 logging 2013-04-03 10:44:13.345492 mon.0 10.252.24.25:6789/0 161 : [INF] pgmap v29095: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:44:13.357240 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos 2013-04-03 10:44:13.357256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 2013-04-03 10:44:13.357273 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:13.368855 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29095 check_down_pgs 2013-04-03 10:44:13.368884 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29095 v29095: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:44:13.368910 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:44:13.369040 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:44:13.369080 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:44:13.369111 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos 2013-04-03 10:44:13.369128 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 2013-04-03 10:44:13.369137 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394090 log 2013-04-03 10:44:13.369153 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:13.369172 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:44:13.392856 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21817 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:44:14.345150 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394090 encode_pending v394091 2013-04-03 10:44:14.375421 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos 2013-04-03 10:44:14.375449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394090 2013-04-03 10:44:14.375465 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos latest full 393798 2013-04-03 10:44:14.375483 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394091 update_from_paxos applying incremental log 394091 2013-04-03 10:44:13.345492 mon.0 10.252.24.25:6789/0 161 : [INF] pgmap v29095: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s 2013-04-03 10:44:14.375531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 check_subs 2013-04-03 10:44:14.375563 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 create_pending v 394092 2013-04-03 10:44:14.375585 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394091 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:44:14.375650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos 2013-04-03 10:44:14.375665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394091 2013-04-03 10:44:14.375684 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:14.400337 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos 2013-04-03 10:44:14.400353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394091 2013-04-03 10:44:14.400370 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:15.594613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140447 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:44:16.941027 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:16.941038 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc5340 from 10.252.24.88:6789/0 2013-04-03 10:44:16.941093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 162 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:44:17.376261 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos 2013-04-03 10:44:17.376282 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394091 2013-04-03 10:44:17.376299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:17.392966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21818 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:44:18.267244 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:18.267254 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 2013-04-03 10:44:18.267305 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 preprocess_query pg_stats(1 pgs tid 31 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:44:18.267329 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 prepare_update pg_stats(1 pgs tid 31 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:44:18.267339 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 prepare_pg_stats pg_stats(1 pgs tid 31 v 605) v1 from osd.2 2013-04-03 10:44:18.267345 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:44:18.317539 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29095 encode_pending v 29096 2013-04-03 10:44:18.346858 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29095 update_from_paxos applying incremental 29096 2013-04-03 10:44:18.346872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 v29096: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1406B/s wr, 0op/s 2013-04-03 10:44:18.346892 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 send_pg_creates to 0 pgs 2013-04-03 10:44:18.346894 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 update_logger 2013-04-03 10:44:18.346904 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 create_pending v 29097 2013-04-03 10:44:18.346912 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29096 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:44:18.346932 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 check_osd_map already seen 605 >= 605 2013-04-03 10:44:18.346936 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 update_logger 2013-04-03 10:44:18.346946 7fcdffcdb700 0 log [INF] : pgmap v29096: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1406B/s wr, 0op/s 2013-04-03 10:44:18.347013 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos 2013-04-03 10:44:18.347032 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394091 2013-04-03 10:44:18.347051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:18.347093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos 2013-04-03 10:44:18.347108 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394091 2013-04-03 10:44:18.347117 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:18.347128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:44:18.347151 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:18.347162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:44:18.347172 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 logging 2013-04-03 10:44:18.346947 mon.0 10.252.24.25:6789/0 162 : [INF] pgmap v29096: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1406B/s wr, 0op/s 2013-04-03 10:44:18.359569 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos 2013-04-03 10:44:18.359586 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394091 2013-04-03 10:44:18.359600 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:18.369284 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29096 check_down_pgs 2013-04-03 10:44:18.369324 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29096 v29096: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1406B/s wr, 0op/s 2013-04-03 10:44:18.369349 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:44:18.369369 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:44:18.369405 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:44:18.369436 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos 2013-04-03 10:44:18.369453 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394091 2013-04-03 10:44:18.369462 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394091 log 2013-04-03 10:44:18.369477 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:18.369497 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:44:19.346501 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394091 encode_pending v394092 2013-04-03 10:44:19.434103 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos 2013-04-03 10:44:19.434134 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos version 394092 summary v 394091 2013-04-03 10:44:19.434148 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos latest full 393798 2013-04-03 10:44:19.434167 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394092 update_from_paxos applying incremental log 394092 2013-04-03 10:44:18.346947 mon.0 10.252.24.25:6789/0 162 : [INF] pgmap v29096: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1406B/s wr, 0op/s 2013-04-03 10:44:19.434216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 check_subs 2013-04-03 10:44:19.434247 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 create_pending v 394093 2013-04-03 10:44:19.434269 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394092 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:44:19.434334 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos 2013-04-03 10:44:19.434350 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos version 394092 summary v 394092 2013-04-03 10:44:19.434368 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:19.445720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos 2013-04-03 10:44:19.445740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos version 394092 summary v 394092 2013-04-03 10:44:19.445758 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:19.594704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140448 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:44:20.941191 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:20.941201 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4840 from 10.252.24.88:6789/0 2013-04-03 10:44:20.941256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 163 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:44:21.049245 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003500 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33117/0) 2013-04-03 10:44:21.393136 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21819 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:44:21.621676 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:21.621686 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 2013-04-03 10:44:21.621742 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 preprocess_query pg_stats(1 pgs tid 50 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:44:21.621766 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 prepare_update pg_stats(1 pgs tid 50 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:44:21.621772 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 prepare_pg_stats pg_stats(1 pgs tid 50 v 605) v1 from osd.1 2013-04-03 10:44:21.621778 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:44:21.672332 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29096 encode_pending v 29097 2013-04-03 10:44:21.780318 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29096 update_from_paxos applying incremental 29097 2013-04-03 10:44:21.780338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 v29097: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:44:21.780356 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 send_pg_creates to 0 pgs 2013-04-03 10:44:21.780360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 update_logger 2013-04-03 10:44:21.780370 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 create_pending v 29098 2013-04-03 10:44:21.780378 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29097 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:44:21.780395 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 check_osd_map already seen 605 >= 605 2013-04-03 10:44:21.780398 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 update_logger 2013-04-03 10:44:21.780408 7fcdffcdb700 0 log [INF] : pgmap v29097: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:44:21.780469 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos 2013-04-03 10:44:21.780488 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos version 394092 summary v 394092 2013-04-03 10:44:21.780510 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:21.780552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos 2013-04-03 10:44:21.780567 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos version 394092 summary v 394092 2013-04-03 10:44:21.780576 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:21.780588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:44:21.780612 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:21.780624 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:44:21.780633 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 logging 2013-04-03 10:44:21.780409 mon.0 10.252.24.25:6789/0 163 : [INF] pgmap v29097: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:44:21.788895 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos 2013-04-03 10:44:21.788913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos version 394092 summary v 394092 2013-04-03 10:44:21.788929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:21.789231 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:21.789239 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:44:21.789264 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 preprocess_query pg_stats(1 pgs tid 30457 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:44:21.789283 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 prepare_update pg_stats(1 pgs tid 30457 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:44:21.789289 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 prepare_pg_stats pg_stats(1 pgs tid 30457 v 605) v1 from osd.3 2013-04-03 10:44:21.789296 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:44:22.735494 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29097 encode_pending v 29098 2013-04-03 10:44:22.736653 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394092 encode_pending v394093 2013-04-03 10:44:22.763575 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29097 update_from_paxos applying incremental 29098 2013-04-03 10:44:22.763594 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 v29098: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s 2013-04-03 10:44:22.763614 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 send_pg_creates to 0 pgs 2013-04-03 10:44:22.763619 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 update_logger 2013-04-03 10:44:22.763629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 create_pending v 29099 2013-04-03 10:44:22.763638 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29098 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:44:22.763655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 check_osd_map already seen 605 >= 605 2013-04-03 10:44:22.763658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 update_logger 2013-04-03 10:44:22.763668 7fcdffcdb700 0 log [INF] : pgmap v29098: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s 2013-04-03 10:44:22.801741 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos 2013-04-03 10:44:22.801765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394092 2013-04-03 10:44:22.801779 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos latest full 393798 2013-04-03 10:44:22.801798 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394093 update_from_paxos applying incremental log 394093 2013-04-03 10:44:21.780409 mon.0 10.252.24.25:6789/0 163 : [INF] pgmap v29097: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:44:22.801851 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 check_subs 2013-04-03 10:44:22.801883 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 create_pending v 394094 2013-04-03 10:44:22.801905 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394093 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:44:22.801938 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos 2013-04-03 10:44:22.801953 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 2013-04-03 10:44:22.801962 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:22.801972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:44:22.801989 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 nothing new 2013-04-03 10:44:22.802040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos 2013-04-03 10:44:22.802055 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 2013-04-03 10:44:22.802074 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:22.821826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos 2013-04-03 10:44:22.821843 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 2013-04-03 10:44:22.821858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:23.369647 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29098 check_down_pgs 2013-04-03 10:44:23.369674 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29098 v29098: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s 2013-04-03 10:44:23.369702 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:44:23.369728 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:44:23.369765 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:44:23.369795 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos 2013-04-03 10:44:23.369814 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 2013-04-03 10:44:23.369823 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394093 log 2013-04-03 10:44:23.369840 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:23.369858 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:44:23.594876 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140449 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:44:24.941284 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:24.941295 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2995b00 from 10.252.24.88:6789/0 2013-04-03 10:44:24.941356 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 164 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:44:25.393217 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21820 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:44:25.802649 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos 2013-04-03 10:44:25.802669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 2013-04-03 10:44:25.802686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:27.595011 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140450 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:44:28.268464 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:28.268473 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:44:28.268523 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 preprocess_query pg_stats(1 pgs tid 32 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:44:28.268547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 prepare_update pg_stats(1 pgs tid 32 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:44:28.268553 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 prepare_pg_stats pg_stats(1 pgs tid 32 v 605) v1 from osd.2 2013-04-03 10:44:28.268559 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:44:28.318745 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29098 encode_pending v 29099 2013-04-03 10:44:28.349957 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29098 update_from_paxos applying incremental 29099 2013-04-03 10:44:28.349978 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29099 v29099: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1386B/s wr, 0op/s 2013-04-03 10:44:28.349997 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29099 send_pg_creates to 0 pgs 2013-04-03 10:44:28.350002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29099 update_logger 2013-04-03 10:44:28.350013 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29099 create_pending v 29100 2013-04-03 10:44:28.350022 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29099 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:44:28.350040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29099 check_osd_map already seen 605 >= 605 2013-04-03 10:44:28.350044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29099 update_logger 2013-04-03 10:44:28.350054 7fcdffcdb700 0 log [INF] : pgmap v29099: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1386B/s wr, 0op/s 2013-04-03 10:44:28.350747 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos 2013-04-03 10:44:28.350767 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 2013-04-03 10:44:28.350784 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:28.350834 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos 2013-04-03 10:44:28.350849 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 2013-04-03 10:44:28.351115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:28.351137 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:44:28.351170 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:28.351183 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:44:28.351194 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 logging 2013-04-03 10:44:22.763669 mon.0 10.252.24.25:6789/0 164 : [INF] pgmap v29098: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s 2013-04-03 10:44:28.351216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 logging 2013-04-03 10:44:28.350056 mon.0 10.252.24.25:6789/0 165 : [INF] pgmap v29099: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1386B/s wr, 0op/s 2013-04-03 10:44:28.362754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos 2013-04-03 10:44:28.362773 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 2013-04-03 10:44:28.362789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:28.369982 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29099 check_down_pgs 2013-04-03 10:44:28.370011 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29099 v29099: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1386B/s wr, 0op/s 2013-04-03 10:44:28.370050 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:44:28.370072 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:44:28.370108 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:44:28.370138 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos 2013-04-03 10:44:28.370155 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 2013-04-03 10:44:28.370166 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394093 log 2013-04-03 10:44:28.370182 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:28.370199 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:44:28.815432 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:44:28.815439 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:44:28.815447 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 2013-04-03 10:44:28.941372 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:28.941384 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29952c0 from 10.252.24.88:6789/0 2013-04-03 10:44:28.941436 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 165 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:44:29.349665 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394093 encode_pending v394094 2013-04-03 10:44:29.393244 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21821 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:44:29.402900 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos 2013-04-03 10:44:29.402927 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394093 2013-04-03 10:44:29.402944 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos latest full 393798 2013-04-03 10:44:29.402964 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos applying incremental log 394094 2013-04-03 10:44:22.763669 mon.0 10.252.24.25:6789/0 164 : [INF] pgmap v29098: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s 2013-04-03 10:44:29.402997 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos applying incremental log 394094 2013-04-03 10:44:28.350056 mon.0 10.252.24.25:6789/0 165 : [INF] pgmap v29099: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1386B/s wr, 0op/s 2013-04-03 10:44:29.403046 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 check_subs 2013-04-03 10:44:29.403080 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 create_pending v 394095 2013-04-03 10:44:29.403103 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:44:29.403166 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos 2013-04-03 10:44:29.403182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 2013-04-03 10:44:29.403202 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:29.428352 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos 2013-04-03 10:44:29.428370 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 2013-04-03 10:44:29.428385 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:31.595153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140451 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:44:32.403777 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos 2013-04-03 10:44:32.403798 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 2013-04-03 10:44:32.403816 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:32.941502 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:32.941513 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2995580 from 10.252.24.88:6789/0 2013-04-03 10:44:32.941561 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 166 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:44:33.370317 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29099 v29099: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1386B/s wr, 0op/s 2013-04-03 10:44:33.370360 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:44:33.370382 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:44:33.370427 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:44:33.370459 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos 2013-04-03 10:44:33.370478 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 2013-04-03 10:44:33.370488 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 log 2013-04-03 10:44:33.370505 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:33.370524 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:44:33.393403 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21822 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:44:35.403905 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos 2013-04-03 10:44:35.403926 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 2013-04-03 10:44:35.403944 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:35.595450 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140452 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:44:36.623418 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:36.623428 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 2013-04-03 10:44:36.623484 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29099 preprocess_query pg_stats(1 pgs tid 51 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:44:36.623509 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29099 prepare_update pg_stats(1 pgs tid 51 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:44:36.623515 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29099 prepare_pg_stats pg_stats(1 pgs tid 51 v 605) v1 from osd.1 2013-04-03 10:44:36.623522 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29099 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:44:36.673706 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29099 encode_pending v 29100 2013-04-03 10:44:36.704855 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29099 update_from_paxos applying incremental 29100 2013-04-03 10:44:36.704873 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 v29100: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1175B/s wr, 0op/s 2013-04-03 10:44:36.704893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 send_pg_creates to 0 pgs 2013-04-03 10:44:36.704897 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 update_logger 2013-04-03 10:44:36.704908 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 create_pending v 29101 2013-04-03 10:44:36.704918 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:44:36.704934 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 check_osd_map already seen 605 >= 605 2013-04-03 10:44:36.704938 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 update_logger 2013-04-03 10:44:36.704955 7fcdffcdb700 0 log [INF] : pgmap v29100: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1175B/s wr, 0op/s 2013-04-03 10:44:36.705018 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos 2013-04-03 10:44:36.705038 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 2013-04-03 10:44:36.705057 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:36.705101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos 2013-04-03 10:44:36.705117 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 2013-04-03 10:44:36.705126 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:36.705138 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:44:36.705164 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:36.705177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:44:36.705189 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 logging 2013-04-03 10:44:36.704958 mon.0 10.252.24.25:6789/0 166 : [INF] pgmap v29100: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1175B/s wr, 0op/s 2013-04-03 10:44:36.716499 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos 2013-04-03 10:44:36.716516 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 2013-04-03 10:44:36.716533 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:36.941643 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:36.941653 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29968c0 from 10.252.24.88:6789/0 2013-04-03 10:44:36.941705 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 167 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:44:37.393484 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21823 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:44:37.704936 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 encode_pending v394095 2013-04-03 10:44:37.737544 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos 2013-04-03 10:44:37.737572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394094 2013-04-03 10:44:37.737587 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos latest full 393798 2013-04-03 10:44:37.737606 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos applying incremental log 394095 2013-04-03 10:44:36.704958 mon.0 10.252.24.25:6789/0 166 : [INF] pgmap v29100: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1175B/s wr, 0op/s 2013-04-03 10:44:37.737664 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 check_subs 2013-04-03 10:44:37.737698 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 create_pending v 394096 2013-04-03 10:44:37.737722 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:44:37.737791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos 2013-04-03 10:44:37.737808 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 2013-04-03 10:44:37.737835 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:37.765941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos 2013-04-03 10:44:37.765958 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 2013-04-03 10:44:37.765975 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:38.370651 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 check_down_pgs 2013-04-03 10:44:38.370687 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 v29100: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1175B/s wr, 0op/s 2013-04-03 10:44:38.370714 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:44:38.370738 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:44:38.370776 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:44:38.370807 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos 2013-04-03 10:44:38.370825 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 2013-04-03 10:44:38.370835 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 log 2013-04-03 10:44:38.370852 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:38.370870 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:44:39.595388 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140453 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:44:40.738446 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos 2013-04-03 10:44:40.738467 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 2013-04-03 10:44:40.738485 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:40.941705 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:40.941714 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ef9580 from 10.252.24.88:6789/0 2013-04-03 10:44:40.941748 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 168 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:44:41.081267 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33121/0) 2013-04-03 10:44:41.393614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21824 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:44:43.371013 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 v29100: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1175B/s wr, 0op/s 2013-04-03 10:44:43.371055 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:44:43.371081 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:44:43.371125 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:44:43.371156 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos 2013-04-03 10:44:43.371176 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 2013-04-03 10:44:43.371185 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 log 2013-04-03 10:44:43.371202 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:43.371222 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:44:43.595493 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140454 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:44:43.738491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos 2013-04-03 10:44:43.738511 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 2013-04-03 10:44:43.738529 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:44.941916 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:44.941926 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ef9840 from 10.252.24.88:6789/0 2013-04-03 10:44:44.941982 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 169 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:44:45.393699 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21825 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:44:46.738538 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos 2013-04-03 10:44:46.738559 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 2013-04-03 10:44:46.738577 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:47.595657 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140455 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:44:48.271050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:48.271059 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 2013-04-03 10:44:48.271103 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 preprocess_query pg_stats(1 pgs tid 33 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:44:48.271127 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 prepare_update pg_stats(1 pgs tid 33 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:44:48.271136 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 prepare_pg_stats pg_stats(1 pgs tid 33 v 605) v1 from osd.2 2013-04-03 10:44:48.271143 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:44:48.321328 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 encode_pending v 29101 2013-04-03 10:44:48.353894 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 update_from_paxos applying incremental 29101 2013-04-03 10:44:48.353911 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 v29101: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s 2013-04-03 10:44:48.353931 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 send_pg_creates to 0 pgs 2013-04-03 10:44:48.353935 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 update_logger 2013-04-03 10:44:48.353946 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 create_pending v 29102 2013-04-03 10:44:48.353958 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:44:48.353975 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 check_osd_map already seen 605 >= 605 2013-04-03 10:44:48.353978 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 update_logger 2013-04-03 10:44:48.353989 7fcdffcdb700 0 log [INF] : pgmap v29101: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s 2013-04-03 10:44:48.354054 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos 2013-04-03 10:44:48.354075 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 2013-04-03 10:44:48.354094 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:48.354137 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos 2013-04-03 10:44:48.354153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 2013-04-03 10:44:48.354162 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:48.354174 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:44:48.354200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:48.354213 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:44:48.354223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 logging 2013-04-03 10:44:48.353990 mon.0 10.252.24.25:6789/0 167 : [INF] pgmap v29101: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s 2013-04-03 10:44:48.368551 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos 2013-04-03 10:44:48.368567 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 2013-04-03 10:44:48.368584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:48.371332 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 check_down_pgs 2013-04-03 10:44:48.371362 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 v29101: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s 2013-04-03 10:44:48.371392 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:44:48.371691 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:44:48.371732 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:44:48.371764 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos 2013-04-03 10:44:48.371782 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 2013-04-03 10:44:48.371791 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 log 2013-04-03 10:44:48.371808 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:48.371825 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:44:48.942014 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:48.942024 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2efa8c0 from 10.252.24.88:6789/0 2013-04-03 10:44:48.942078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 170 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:44:49.348412 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 encode_pending v394096 2013-04-03 10:44:49.377073 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos 2013-04-03 10:44:49.377098 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394095 2013-04-03 10:44:49.377114 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos latest full 393798 2013-04-03 10:44:49.377133 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos applying incremental log 394096 2013-04-03 10:44:48.353990 mon.0 10.252.24.25:6789/0 167 : [INF] pgmap v29101: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s 2013-04-03 10:44:49.377180 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 check_subs 2013-04-03 10:44:49.377212 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 create_pending v 394097 2013-04-03 10:44:49.377235 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:44:49.377301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos 2013-04-03 10:44:49.377328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 2013-04-03 10:44:49.377348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:49.393796 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21826 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:44:49.401303 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos 2013-04-03 10:44:49.401328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 2013-04-03 10:44:49.401345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:51.097219 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33122/0) 2013-04-03 10:44:51.595758 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140456 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:44:52.377911 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos 2013-04-03 10:44:52.377932 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 2013-04-03 10:44:52.377949 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:52.942102 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:52.942113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ef92c0 from 10.252.24.88:6789/0 2013-04-03 10:44:52.942160 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 171 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:44:53.371948 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 v29101: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s 2013-04-03 10:44:53.371990 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:44:53.372011 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:44:53.372055 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:44:53.372086 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos 2013-04-03 10:44:53.372106 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 2013-04-03 10:44:53.372238 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 log 2013-04-03 10:44:53.372261 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:53.372282 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:44:53.393917 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21827 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:44:55.378139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos 2013-04-03 10:44:55.378160 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 2013-04-03 10:44:55.378178 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:55.595871 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140457 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:44:56.625881 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:56.625893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:44:56.625945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 preprocess_query pg_stats(1 pgs tid 52 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:44:56.625970 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 prepare_update pg_stats(1 pgs tid 52 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:44:56.625977 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 prepare_pg_stats pg_stats(1 pgs tid 52 v 605) v1 from osd.1 2013-04-03 10:44:56.625984 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:44:56.676192 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 encode_pending v 29102 2013-04-03 10:44:56.946526 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 update_from_paxos applying incremental 29102 2013-04-03 10:44:56.946545 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 v29102: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s 2013-04-03 10:44:56.946565 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 send_pg_creates to 0 pgs 2013-04-03 10:44:56.946569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 update_logger 2013-04-03 10:44:56.946580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 create_pending v 29103 2013-04-03 10:44:56.946590 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:44:56.946608 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 check_osd_map already seen 605 >= 605 2013-04-03 10:44:56.946612 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 update_logger 2013-04-03 10:44:56.946623 7fcdffcdb700 0 log [INF] : pgmap v29102: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s 2013-04-03 10:44:56.946689 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos 2013-04-03 10:44:56.946709 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 2013-04-03 10:44:56.946728 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:56.946772 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos 2013-04-03 10:44:56.946790 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 2013-04-03 10:44:56.946799 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:56.946812 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:44:56.946837 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:56.946850 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:44:56.946860 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 logging 2013-04-03 10:44:56.946624 mon.0 10.252.24.25:6789/0 168 : [INF] pgmap v29102: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s 2013-04-03 10:44:56.958061 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos 2013-04-03 10:44:56.958078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 2013-04-03 10:44:56.958096 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:56.958369 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:56.958378 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 2013-04-03 10:44:56.958406 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 preprocess_query pg_stats(1 pgs tid 30458 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:44:56.958427 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 prepare_update pg_stats(1 pgs tid 30458 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:44:56.958433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 prepare_pg_stats pg_stats(1 pgs tid 30458 v 605) v1 from osd.3 2013-04-03 10:44:56.958439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:44:56.958471 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:56.958479 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ef9dc0 from 10.252.24.88:6789/0 2013-04-03 10:44:56.958505 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 172 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:44:57.394080 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21828 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:44:57.946308 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 encode_pending v 29103 2013-04-03 10:44:57.947403 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 encode_pending v394097 2013-04-03 10:44:57.990947 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 update_from_paxos applying incremental 29103 2013-04-03 10:44:57.990967 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 v29103: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 531B/s wr, 0op/s 2013-04-03 10:44:57.990987 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 send_pg_creates to 0 pgs 2013-04-03 10:44:57.990991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 update_logger 2013-04-03 10:44:57.991002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 create_pending v 29104 2013-04-03 10:44:57.991011 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:44:57.991030 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 check_osd_map already seen 605 >= 605 2013-04-03 10:44:57.991034 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 update_logger 2013-04-03 10:44:57.991045 7fcdffcdb700 0 log [INF] : pgmap v29103: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 531B/s wr, 0op/s 2013-04-03 10:44:58.041955 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos 2013-04-03 10:44:58.041981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos version 394097 summary v 394096 2013-04-03 10:44:58.041995 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos latest full 393798 2013-04-03 10:44:58.042015 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos applying incremental log 394097 2013-04-03 10:44:56.946624 mon.0 10.252.24.25:6789/0 168 : [INF] pgmap v29102: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s 2013-04-03 10:44:58.042060 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 check_subs 2013-04-03 10:44:58.042092 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 create_pending v 394098 2013-04-03 10:44:58.042115 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:44:58.042151 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos 2013-04-03 10:44:58.042167 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos version 394097 summary v 394097 2013-04-03 10:44:58.042177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:58.042189 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:44:58.042209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 nothing new 2013-04-03 10:44:58.042260 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos 2013-04-03 10:44:58.042276 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos version 394097 summary v 394097 2013-04-03 10:44:58.042295 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:58.052964 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos 2013-04-03 10:44:58.052981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos version 394097 summary v 394097 2013-04-03 10:44:58.052998 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:58.272186 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:44:58.272195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:44:58.272226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 preprocess_query pg_stats(1 pgs tid 34 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:44:58.272248 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 prepare_update pg_stats(1 pgs tid 34 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:44:58.272257 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 prepare_pg_stats pg_stats(1 pgs tid 34 v 605) v1 from osd.2 2013-04-03 10:44:58.272263 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:44:58.372393 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 check_down_pgs 2013-04-03 10:44:58.372422 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 v29103: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 531B/s wr, 0op/s 2013-04-03 10:44:58.372451 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:44:58.372473 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:44:58.372515 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:44:58.372547 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos 2013-04-03 10:44:58.372564 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos version 394097 summary v 394097 2013-04-03 10:44:58.372574 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 log 2013-04-03 10:44:58.372590 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:44:58.372607 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:44:58.815522 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) 2013-04-03 10:44:58.815566 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader 2013-04-03 10:44:59.041046 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29103 encode_pending v 29104 2013-04-03 10:44:59.069370 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29103 update_from_paxos applying incremental 29104 2013-04-03 10:44:59.069389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 v29104: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 5629B/s wr, 0op/s 2013-04-03 10:44:59.069408 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 send_pg_creates to 0 pgs 2013-04-03 10:44:59.069411 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 update_logger 2013-04-03 10:44:59.069421 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 create_pending v 29105 2013-04-03 10:44:59.069430 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29104 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:44:59.069448 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 check_osd_map already seen 605 >= 605 2013-04-03 10:44:59.069451 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 update_logger 2013-04-03 10:44:59.069461 7fcdffcdb700 0 log [INF] : pgmap v29104: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 5629B/s wr, 0op/s 2013-04-03 10:44:59.070638 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 update_from_paxos 2013-04-03 10:44:59.070658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 update_from_paxos version 394097 summary v 394097 2013-04-03 10:44:59.070676 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:59.070721 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 update_from_paxos 2013-04-03 10:44:59.070736 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 update_from_paxos version 394097 summary v 394097 2013-04-03 10:44:59.070745 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:59.070757 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:44:59.070788 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:44:59.070801 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:44:59.070812 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 logging 2013-04-03 10:44:57.991052 mon.0 10.252.24.25:6789/0 169 : [INF] pgmap v29103: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 531B/s wr, 0op/s 2013-04-03 10:44:59.070833 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 logging 2013-04-03 10:44:59.069462 mon.0 10.252.24.25:6789/0 170 : [INF] pgmap v29104: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 5629B/s wr, 0op/s 2013-04-03 10:44:59.094157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 update_from_paxos 2013-04-03 10:44:59.094173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 update_from_paxos version 394097 summary v 394097 2013-04-03 10:44:59.094190 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:44:59.596010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140458 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:00.069127 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394097 encode_pending v394098 2013-04-03 10:45:00.092587 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos 2013-04-03 10:45:00.092614 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394097 2013-04-03 10:45:00.092629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos latest full 393798 2013-04-03 10:45:00.093278 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394098 update_from_paxos applying incremental log 394098 2013-04-03 10:44:57.991052 mon.0 10.252.24.25:6789/0 169 : [INF] pgmap v29103: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 531B/s wr, 0op/s 2013-04-03 10:45:00.093315 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394098 update_from_paxos applying incremental log 394098 2013-04-03 10:44:59.069462 mon.0 10.252.24.25:6789/0 170 : [INF] pgmap v29104: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 5629B/s wr, 0op/s 2013-04-03 10:45:00.093353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 check_subs 2013-04-03 10:45:00.093385 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 create_pending v 394099 2013-04-03 10:45:00.093407 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394098 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:45:00.093472 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos 2013-04-03 10:45:00.093488 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 2013-04-03 10:45:00.093506 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:00.105477 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos 2013-04-03 10:45:00.105494 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 2013-04-03 10:45:00.105508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:00.942366 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:00.942376 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2efab80 from 10.252.24.88:6789/0 2013-04-03 10:45:00.942428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 173 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:45:01.113188 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3062280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33164/0) 2013-04-03 10:45:01.394173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21829 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:45:03.093480 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos 2013-04-03 10:45:03.093500 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 2013-04-03 10:45:03.093517 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:03.372718 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29104 check_down_pgs 2013-04-03 10:45:03.372747 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29104 v29104: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 5629B/s wr, 0op/s 2013-04-03 10:45:03.372774 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:45:03.372793 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:45:03.372829 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:45:03.372860 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos 2013-04-03 10:45:03.372876 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 2013-04-03 10:45:03.372885 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394098 log 2013-04-03 10:45:03.372900 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:03.372917 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:45:03.596140 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140459 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:04.942495 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:04.942505 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83000 from 10.252.24.88:6789/0 2013-04-03 10:45:04.942558 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 174 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:45:05.394258 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21830 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:45:06.093518 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos 2013-04-03 10:45:06.093538 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 2013-04-03 10:45:06.093557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:06.626954 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:06.626964 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 2013-04-03 10:45:06.627016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 preprocess_query pg_stats(1 pgs tid 53 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:06.627041 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 prepare_update pg_stats(1 pgs tid 53 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:06.627048 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 prepare_pg_stats pg_stats(1 pgs tid 53 v 605) v1 from osd.1 2013-04-03 10:45:06.627056 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:45:06.677242 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29104 encode_pending v 29105 2013-04-03 10:45:06.716879 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29104 update_from_paxos applying incremental 29105 2013-04-03 10:45:06.716894 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 v29105: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1876B/s wr, 0op/s 2013-04-03 10:45:06.716914 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 send_pg_creates to 0 pgs 2013-04-03 10:45:06.716917 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 update_logger 2013-04-03 10:45:06.716927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 create_pending v 29106 2013-04-03 10:45:06.716936 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29105 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:06.716952 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 check_osd_map already seen 605 >= 605 2013-04-03 10:45:06.716955 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 update_logger 2013-04-03 10:45:06.716965 7fcdffcdb700 0 log [INF] : pgmap v29105: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1876B/s wr, 0op/s 2013-04-03 10:45:06.717023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos 2013-04-03 10:45:06.717044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 2013-04-03 10:45:06.717063 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:06.717105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos 2013-04-03 10:45:06.717120 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 2013-04-03 10:45:06.717129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:06.717141 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:45:06.717165 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:06.717177 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:45:06.717186 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 logging 2013-04-03 10:45:06.716966 mon.0 10.252.24.25:6789/0 171 : [INF] pgmap v29105: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1876B/s wr, 0op/s 2013-04-03 10:45:06.741315 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos 2013-04-03 10:45:06.741333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 2013-04-03 10:45:06.741349 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:07.596251 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140460 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:07.716150 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394098 encode_pending v394099 2013-04-03 10:45:07.739704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos 2013-04-03 10:45:07.739731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394098 2013-04-03 10:45:07.739746 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos latest full 393798 2013-04-03 10:45:07.739765 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394099 update_from_paxos applying incremental log 394099 2013-04-03 10:45:06.716966 mon.0 10.252.24.25:6789/0 171 : [INF] pgmap v29105: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1876B/s wr, 0op/s 2013-04-03 10:45:07.739815 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 check_subs 2013-04-03 10:45:07.739847 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 create_pending v 394100 2013-04-03 10:45:07.739869 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394099 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:45:07.739936 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos 2013-04-03 10:45:07.739951 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 2013-04-03 10:45:07.739969 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:07.752612 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos 2013-04-03 10:45:07.752629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 2013-04-03 10:45:07.752646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:08.373060 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29105 check_down_pgs 2013-04-03 10:45:08.373093 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29105 v29105: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1876B/s wr, 0op/s 2013-04-03 10:45:08.373120 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:45:08.373142 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:45:08.373179 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:45:08.373210 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos 2013-04-03 10:45:08.373229 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 2013-04-03 10:45:08.373238 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394099 log 2013-04-03 10:45:08.373255 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:08.373274 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:45:08.942742 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:08.942751 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83580 from 10.252.24.88:6789/0 2013-04-03 10:45:08.942804 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 175 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:45:09.394424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21831 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:45:10.740694 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos 2013-04-03 10:45:10.740714 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 2013-04-03 10:45:10.740731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:11.129221 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33165/0) 2013-04-03 10:45:11.596434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140461 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:12.942752 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:12.942762 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a84340 from 10.252.24.88:6789/0 2013-04-03 10:45:12.942815 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 176 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:45:13.273959 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:13.273967 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:45:13.274000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 preprocess_query pg_stats(1 pgs tid 35 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:13.274023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 prepare_update pg_stats(1 pgs tid 35 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:13.274029 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 prepare_pg_stats pg_stats(1 pgs tid 35 v 605) v1 from osd.2 2013-04-03 10:45:13.274035 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:45:13.324221 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29105 encode_pending v 29106 2013-04-03 10:45:13.355665 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29105 update_from_paxos applying incremental 29106 2013-04-03 10:45:13.355680 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 v29106: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 50471B/s wr, 0op/s 2013-04-03 10:45:13.355699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 send_pg_creates to 0 pgs 2013-04-03 10:45:13.355702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 update_logger 2013-04-03 10:45:13.355714 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 create_pending v 29107 2013-04-03 10:45:13.355723 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29106 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:13.355737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 check_osd_map already seen 605 >= 605 2013-04-03 10:45:13.355740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 update_logger 2013-04-03 10:45:13.355750 7fcdffcdb700 0 log [INF] : pgmap v29106: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 50471B/s wr, 0op/s 2013-04-03 10:45:13.355810 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos 2013-04-03 10:45:13.355829 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 2013-04-03 10:45:13.355848 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:13.355891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos 2013-04-03 10:45:13.355905 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 2013-04-03 10:45:13.355914 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:13.355926 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:45:13.355951 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:13.355965 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:45:13.355976 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 logging 2013-04-03 10:45:13.355752 mon.0 10.252.24.25:6789/0 172 : [INF] pgmap v29106: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 50471B/s wr, 0op/s 2013-04-03 10:45:13.373387 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29106 check_down_pgs 2013-04-03 10:45:13.373410 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29106 v29106: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 50471B/s wr, 0op/s 2013-04-03 10:45:13.373431 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:45:13.373451 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:45:13.373486 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:45:13.373516 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos 2013-04-03 10:45:13.373533 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 2013-04-03 10:45:13.373542 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394099 log 2013-04-03 10:45:13.373565 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:13.373583 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:45:13.380498 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos 2013-04-03 10:45:13.380515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 2013-04-03 10:45:13.380530 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:13.394434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21832 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:45:14.355476 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394099 encode_pending v394100 2013-04-03 10:45:14.380749 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos 2013-04-03 10:45:14.380776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394099 2013-04-03 10:45:14.380791 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos latest full 393798 2013-04-03 10:45:14.380808 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394100 update_from_paxos applying incremental log 394100 2013-04-03 10:45:13.355752 mon.0 10.252.24.25:6789/0 172 : [INF] pgmap v29106: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 50471B/s wr, 0op/s 2013-04-03 10:45:14.380853 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 check_subs 2013-04-03 10:45:14.380885 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 create_pending v 394101 2013-04-03 10:45:14.380907 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394100 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:45:14.380975 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos 2013-04-03 10:45:14.380991 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394100 2013-04-03 10:45:14.381009 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:14.391679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos 2013-04-03 10:45:14.391695 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394100 2013-04-03 10:45:14.391711 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:15.596524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140462 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:16.943179 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:16.943191 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83dc0 from 10.252.24.88:6789/0 2013-04-03 10:45:16.943244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 177 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:45:17.381666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos 2013-04-03 10:45:17.381686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394100 2013-04-03 10:45:17.381703 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:17.394531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21833 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:45:18.274636 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:18.274646 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:45:18.274698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 preprocess_query pg_stats(1 pgs tid 36 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:18.274722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 prepare_update pg_stats(1 pgs tid 36 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:18.274728 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 prepare_pg_stats pg_stats(1 pgs tid 36 v 605) v1 from osd.2 2013-04-03 10:45:18.274734 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:45:18.324918 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29106 encode_pending v 29107 2013-04-03 10:45:18.354685 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29106 update_from_paxos applying incremental 29107 2013-04-03 10:45:18.354706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 v29107: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 61891B/s wr, 0op/s 2013-04-03 10:45:18.354725 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 send_pg_creates to 0 pgs 2013-04-03 10:45:18.354729 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 update_logger 2013-04-03 10:45:18.354740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 create_pending v 29108 2013-04-03 10:45:18.354749 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29107 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:18.354765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 check_osd_map already seen 605 >= 605 2013-04-03 10:45:18.354767 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 update_logger 2013-04-03 10:45:18.354910 7fcdffcdb700 0 log [INF] : pgmap v29107: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 61891B/s wr, 0op/s 2013-04-03 10:45:18.354983 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos 2013-04-03 10:45:18.355003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394100 2013-04-03 10:45:18.355021 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:18.355064 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos 2013-04-03 10:45:18.355079 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394100 2013-04-03 10:45:18.355089 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:18.355102 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:45:18.355126 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:18.355138 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:45:18.355148 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 logging 2013-04-03 10:45:18.354916 mon.0 10.252.24.25:6789/0 173 : [INF] pgmap v29107: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 61891B/s wr, 0op/s 2013-04-03 10:45:18.367379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos 2013-04-03 10:45:18.367396 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394100 2013-04-03 10:45:18.367411 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:18.373694 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29107 check_down_pgs 2013-04-03 10:45:18.373724 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29107 v29107: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 61891B/s wr, 0op/s 2013-04-03 10:45:18.373754 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:45:18.373774 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:45:18.373811 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:45:18.373841 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos 2013-04-03 10:45:18.373858 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394100 2013-04-03 10:45:18.373867 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394100 log 2013-04-03 10:45:18.373883 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:18.373900 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:45:19.354387 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394100 encode_pending v394101 2013-04-03 10:45:19.475242 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos 2013-04-03 10:45:19.475269 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos version 394101 summary v 394100 2013-04-03 10:45:19.475285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos latest full 393798 2013-04-03 10:45:19.475304 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394101 update_from_paxos applying incremental log 394101 2013-04-03 10:45:18.354916 mon.0 10.252.24.25:6789/0 173 : [INF] pgmap v29107: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 61891B/s wr, 0op/s 2013-04-03 10:45:19.475355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 check_subs 2013-04-03 10:45:19.475387 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 create_pending v 394102 2013-04-03 10:45:19.475409 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394101 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:45:19.475475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos 2013-04-03 10:45:19.475491 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos version 394101 summary v 394101 2013-04-03 10:45:19.475509 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:19.486776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos 2013-04-03 10:45:19.486792 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos version 394101 summary v 394101 2013-04-03 10:45:19.486807 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:19.596666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140463 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:20.943281 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:20.943291 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae8c0 from 10.252.24.88:6789/0 2013-04-03 10:45:20.943344 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 178 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:45:21.145238 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33166/0) 2013-04-03 10:45:21.394668 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21834 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:45:21.628853 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:21.628862 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:45:21.628914 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 preprocess_query pg_stats(1 pgs tid 54 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:21.628939 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 prepare_update pg_stats(1 pgs tid 54 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:21.628945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 prepare_pg_stats pg_stats(1 pgs tid 54 v 605) v1 from osd.1 2013-04-03 10:45:21.628951 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:45:21.679137 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29107 encode_pending v 29108 2013-04-03 10:45:21.712670 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29107 update_from_paxos applying incremental 29108 2013-04-03 10:45:21.712686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 v29108: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:45:21.712706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 send_pg_creates to 0 pgs 2013-04-03 10:45:21.712712 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 update_logger 2013-04-03 10:45:21.712723 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 create_pending v 29109 2013-04-03 10:45:21.712732 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29108 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:21.712748 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 check_osd_map already seen 605 >= 605 2013-04-03 10:45:21.712751 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 update_logger 2013-04-03 10:45:21.712761 7fcdffcdb700 0 log [INF] : pgmap v29108: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:45:21.712819 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos 2013-04-03 10:45:21.712839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos version 394101 summary v 394101 2013-04-03 10:45:21.712858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:21.712900 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos 2013-04-03 10:45:21.712915 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos version 394101 summary v 394101 2013-04-03 10:45:21.712924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:21.712936 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:45:21.712962 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:21.712975 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:45:21.712986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 logging 2013-04-03 10:45:21.712762 mon.0 10.252.24.25:6789/0 174 : [INF] pgmap v29108: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:45:21.724742 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos 2013-04-03 10:45:21.724765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos version 394101 summary v 394101 2013-04-03 10:45:21.724781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:21.795086 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:21.795094 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 2013-04-03 10:45:21.795123 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 preprocess_query pg_stats(1 pgs tid 30459 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:45:21.795144 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 prepare_update pg_stats(1 pgs tid 30459 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:45:21.795149 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 prepare_pg_stats pg_stats(1 pgs tid 30459 v 605) v1 from osd.3 2013-04-03 10:45:21.795154 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:45:22.712428 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29108 encode_pending v 29109 2013-04-03 10:45:22.713943 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394101 encode_pending v394102 2013-04-03 10:45:22.743618 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29108 update_from_paxos applying incremental 29109 2013-04-03 10:45:22.743643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29109 v29109: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1166B/s wr, 0op/s 2013-04-03 10:45:22.743665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29109 send_pg_creates to 0 pgs 2013-04-03 10:45:22.743669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29109 update_logger 2013-04-03 10:45:22.743679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29109 create_pending v 29110 2013-04-03 10:45:22.743690 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29109 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:45:22.743875 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29109 check_osd_map already seen 605 >= 605 2013-04-03 10:45:22.743881 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29109 update_logger 2013-04-03 10:45:22.743892 7fcdffcdb700 0 log [INF] : pgmap v29109: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1166B/s wr, 0op/s 2013-04-03 10:45:22.768826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos 2013-04-03 10:45:22.768851 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394101 2013-04-03 10:45:22.768865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos latest full 393798 2013-04-03 10:45:22.768883 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394102 update_from_paxos applying incremental log 394102 2013-04-03 10:45:21.712762 mon.0 10.252.24.25:6789/0 174 : [INF] pgmap v29108: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:45:22.768925 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 check_subs 2013-04-03 10:45:22.768956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 create_pending v 394103 2013-04-03 10:45:22.768978 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394102 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:45:22.769010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos 2013-04-03 10:45:22.769025 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394102 2013-04-03 10:45:22.769034 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:22.769045 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:45:22.769063 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 nothing new 2013-04-03 10:45:22.769113 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos 2013-04-03 10:45:22.769128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394102 2013-04-03 10:45:22.769146 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:22.782278 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos 2013-04-03 10:45:22.782294 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394102 2013-04-03 10:45:22.782310 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:23.374041 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29109 check_down_pgs 2013-04-03 10:45:23.374069 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29109 v29109: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1166B/s wr, 0op/s 2013-04-03 10:45:23.374096 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:45:23.374120 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:45:23.374156 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:45:23.374187 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos 2013-04-03 10:45:23.374206 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394102 2013-04-03 10:45:23.374215 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394102 log 2013-04-03 10:45:23.374232 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:23.374251 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:45:23.596789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140464 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:24.943430 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:24.943440 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2efa340 from 10.252.24.88:6789/0 2013-04-03 10:45:24.943992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 179 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:45:25.394820 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21835 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:45:25.769823 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos 2013-04-03 10:45:25.769842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394102 2013-04-03 10:45:25.769859 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:27.596913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140465 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:28.374398 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29109 v29109: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1166B/s wr, 0op/s 2013-04-03 10:45:28.374443 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:45:28.374466 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:45:28.374508 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:45:28.374539 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos 2013-04-03 10:45:28.374557 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394102 2013-04-03 10:45:28.374916 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394102 log 2013-04-03 10:45:28.374941 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:28.374960 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:45:28.769906 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos 2013-04-03 10:45:28.769925 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394102 2013-04-03 10:45:28.769943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:28.871396 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:45:28.871401 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:45:28.871408 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 2013-04-03 10:45:28.943605 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:28.943614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ef9b00 from 10.252.24.88:6789/0 2013-04-03 10:45:28.943647 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 180 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:45:29.394921 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21836 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:45:31.597054 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140466 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:31.630179 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:31.630188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:45:31.630221 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29109 preprocess_query pg_stats(1 pgs tid 55 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:31.630246 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29109 prepare_update pg_stats(1 pgs tid 55 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:31.630252 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29109 prepare_pg_stats pg_stats(1 pgs tid 55 v 605) v1 from osd.1 2013-04-03 10:45:31.630259 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29109 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:45:31.680445 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29109 encode_pending v 29110 2013-04-03 10:45:31.887497 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29109 update_from_paxos applying incremental 29110 2013-04-03 10:45:31.887514 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 v29110: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 511B/s wr, 0op/s 2013-04-03 10:45:31.887534 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 send_pg_creates to 0 pgs 2013-04-03 10:45:31.887538 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 update_logger 2013-04-03 10:45:31.887549 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 create_pending v 29111 2013-04-03 10:45:31.887563 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:31.887579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 check_osd_map already seen 605 >= 605 2013-04-03 10:45:31.887583 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 update_logger 2013-04-03 10:45:31.887593 7fcdffcdb700 0 log [INF] : pgmap v29110: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 511B/s wr, 0op/s 2013-04-03 10:45:31.887656 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 update_from_paxos 2013-04-03 10:45:31.887676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 update_from_paxos version 394102 summary v 394102 2013-04-03 10:45:31.887695 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:31.887739 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 update_from_paxos 2013-04-03 10:45:31.887754 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 update_from_paxos version 394102 summary v 394102 2013-04-03 10:45:31.887764 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:31.887777 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:45:31.887809 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:31.887825 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:45:31.887836 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 logging 2013-04-03 10:45:22.743893 mon.0 10.252.24.25:6789/0 175 : [INF] pgmap v29109: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1166B/s wr, 0op/s 2013-04-03 10:45:31.887860 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 logging 2013-04-03 10:45:31.887595 mon.0 10.252.24.25:6789/0 176 : [INF] pgmap v29110: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 511B/s wr, 0op/s 2013-04-03 10:45:31.913273 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 update_from_paxos 2013-04-03 10:45:31.913290 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 update_from_paxos version 394102 summary v 394102 2013-04-03 10:45:31.913307 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:32.887287 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 encode_pending v394103 2013-04-03 10:45:32.920417 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos 2013-04-03 10:45:32.920443 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos version 394103 summary v 394102 2013-04-03 10:45:32.920457 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos latest full 393798 2013-04-03 10:45:32.920476 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos applying incremental log 394103 2013-04-03 10:45:22.743893 mon.0 10.252.24.25:6789/0 175 : [INF] pgmap v29109: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1166B/s wr, 0op/s 2013-04-03 10:45:32.920507 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos applying incremental log 394103 2013-04-03 10:45:31.887595 mon.0 10.252.24.25:6789/0 176 : [INF] pgmap v29110: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 511B/s wr, 0op/s 2013-04-03 10:45:32.920548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 check_subs 2013-04-03 10:45:32.920580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 create_pending v 394104 2013-04-03 10:45:32.920603 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:45:32.920670 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos 2013-04-03 10:45:32.920686 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos version 394103 summary v 394103 2013-04-03 10:45:32.920705 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:32.933116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos 2013-04-03 10:45:32.933133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos version 394103 summary v 394103 2013-04-03 10:45:32.933150 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:32.943539 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:32.943548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2145b00 from 10.252.24.88:6789/0 2013-04-03 10:45:32.943577 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 181 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:45:33.276603 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:33.276614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 2013-04-03 10:45:33.276667 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 preprocess_query pg_stats(1 pgs tid 37 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:33.276692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 prepare_update pg_stats(1 pgs tid 37 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:33.276698 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 prepare_pg_stats pg_stats(1 pgs tid 37 v 605) v1 from osd.2 2013-04-03 10:45:33.276705 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:45:33.375074 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 check_down_pgs 2013-04-03 10:45:33.375103 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 v29110: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 511B/s wr, 0op/s 2013-04-03 10:45:33.375137 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:45:33.375160 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:45:33.375197 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:45:33.375228 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos 2013-04-03 10:45:33.375247 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos version 394103 summary v 394103 2013-04-03 10:45:33.375257 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 log 2013-04-03 10:45:33.375274 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:33.375293 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:45:33.395024 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21837 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:45:33.919695 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 encode_pending v 29111 2013-04-03 10:45:33.945315 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 update_from_paxos applying incremental 29111 2013-04-03 10:45:33.945336 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 v29111: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1461B/s wr, 0op/s 2013-04-03 10:45:33.945356 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 send_pg_creates to 0 pgs 2013-04-03 10:45:33.945361 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 update_logger 2013-04-03 10:45:33.945371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 create_pending v 29112 2013-04-03 10:45:33.945381 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:33.945399 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 check_osd_map already seen 605 >= 605 2013-04-03 10:45:33.945402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 update_logger 2013-04-03 10:45:33.945417 7fcdffcdb700 0 log [INF] : pgmap v29111: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1461B/s wr, 0op/s 2013-04-03 10:45:33.945482 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos 2013-04-03 10:45:33.945501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos version 394103 summary v 394103 2013-04-03 10:45:33.945521 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:33.945566 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos 2013-04-03 10:45:33.945582 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos version 394103 summary v 394103 2013-04-03 10:45:33.945591 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:33.945603 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:45:33.945628 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:33.945640 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:45:33.945651 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 logging 2013-04-03 10:45:33.945418 mon.0 10.252.24.25:6789/0 177 : [INF] pgmap v29111: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1461B/s wr, 0op/s 2013-04-03 10:45:33.970862 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos 2013-04-03 10:45:33.970879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos version 394103 summary v 394103 2013-04-03 10:45:33.970897 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:34.945072 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 encode_pending v394104 2013-04-03 10:45:34.996238 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos 2013-04-03 10:45:34.996265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos version 394104 summary v 394103 2013-04-03 10:45:34.996281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos latest full 393798 2013-04-03 10:45:34.996301 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos applying incremental log 394104 2013-04-03 10:45:33.945418 mon.0 10.252.24.25:6789/0 177 : [INF] pgmap v29111: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1461B/s wr, 0op/s 2013-04-03 10:45:34.996349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 check_subs 2013-04-03 10:45:34.996382 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 create_pending v 394105 2013-04-03 10:45:34.996405 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:45:34.996473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos 2013-04-03 10:45:34.996490 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos version 394104 summary v 394104 2013-04-03 10:45:34.996509 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:35.008247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos 2013-04-03 10:45:35.008264 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos version 394104 summary v 394104 2013-04-03 10:45:35.008281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:35.530810 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:35.530821 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 2013-04-03 10:45:35.530874 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 preprocess_query pg_stats(0 pgs tid 63 v 605) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:45:35.530899 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 prepare_update pg_stats(0 pgs tid 63 v 605) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:45:35.530906 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 prepare_pg_stats pg_stats(0 pgs tid 63 v 605) v1 from osd.4 2013-04-03 10:45:35.530913 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 got osd.4 osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[])) 2013-04-03 10:45:35.597150 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140467 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:35.996082 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 encode_pending v 29112 2013-04-03 10:45:36.175832 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 update_from_paxos applying incremental 29112 2013-04-03 10:45:36.175845 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 v29112: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 2847B/s wr, 0op/s 2013-04-03 10:45:36.175863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 send_pg_creates to 0 pgs 2013-04-03 10:45:36.175870 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 update_logger 2013-04-03 10:45:36.175880 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 create_pending v 29113 2013-04-03 10:45:36.175890 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 _updated_stats for osd.4 10.252.24.32:6800/13473 2013-04-03 10:45:36.175908 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 check_osd_map already seen 605 >= 605 2013-04-03 10:45:36.175911 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 update_logger 2013-04-03 10:45:36.175922 7fcdffcdb700 0 log [INF] : pgmap v29112: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 2847B/s wr, 0op/s 2013-04-03 10:45:36.175985 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos 2013-04-03 10:45:36.176005 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos version 394104 summary v 394104 2013-04-03 10:45:36.176024 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:36.176070 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos 2013-04-03 10:45:36.176086 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos version 394104 summary v 394104 2013-04-03 10:45:36.176095 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:36.176107 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:45:36.176136 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:36.176149 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:45:36.176160 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 logging 2013-04-03 10:45:36.175923 mon.0 10.252.24.25:6789/0 178 : [INF] pgmap v29112: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 2847B/s wr, 0op/s 2013-04-03 10:45:36.200930 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos 2013-04-03 10:45:36.200947 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos version 394104 summary v 394104 2013-04-03 10:45:36.200962 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:36.855582 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:36.855593 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 2013-04-03 10:45:36.855645 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 preprocess_query pg_stats(0 pgs tid 412 v 605) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:45:36.855670 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 prepare_update pg_stats(0 pgs tid 412 v 605) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:45:36.855676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 prepare_pg_stats pg_stats(0 pgs tid 412 v 605) v1 from osd.0 2013-04-03 10:45:36.855683 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 got osd.0 osd_stat(1141 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1141 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:45:36.943761 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:36.943769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x21452c0 from 10.252.24.88:6789/0 2013-04-03 10:45:36.943804 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 182 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:45:37.175122 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 encode_pending v 29113 2013-04-03 10:45:37.176404 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 encode_pending v394105 2013-04-03 10:45:37.235882 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 update_from_paxos applying incremental 29113 2013-04-03 10:45:37.235895 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 v29113: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:45:37.235912 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 send_pg_creates to 0 pgs 2013-04-03 10:45:37.235916 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 update_logger 2013-04-03 10:45:37.235927 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 create_pending v 29114 2013-04-03 10:45:37.235937 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:45:37.235954 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 check_osd_map already seen 605 >= 605 2013-04-03 10:45:37.235958 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 update_logger 2013-04-03 10:45:37.235968 7fcdffcdb700 0 log [INF] : pgmap v29113: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:45:37.260803 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos 2013-04-03 10:45:37.260828 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394104 2013-04-03 10:45:37.260844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos latest full 393798 2013-04-03 10:45:37.260863 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos applying incremental log 394105 2013-04-03 10:45:36.175923 mon.0 10.252.24.25:6789/0 178 : [INF] pgmap v29112: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 2847B/s wr, 0op/s 2013-04-03 10:45:37.260910 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 check_subs 2013-04-03 10:45:37.260941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 create_pending v 394106 2013-04-03 10:45:37.260964 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:45:37.260996 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos 2013-04-03 10:45:37.261012 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394105 2013-04-03 10:45:37.261020 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:37.261031 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:45:37.261049 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 nothing new 2013-04-03 10:45:37.261100 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos 2013-04-03 10:45:37.261118 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394105 2013-04-03 10:45:37.261136 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:37.274032 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos 2013-04-03 10:45:37.274049 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394105 2013-04-03 10:45:37.274065 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:37.395156 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21838 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:45:38.277139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:38.277152 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:45:38.277216 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 preprocess_query pg_stats(1 pgs tid 38 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:38.277241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 prepare_update pg_stats(1 pgs tid 38 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:38.277247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 prepare_pg_stats pg_stats(1 pgs tid 38 v 605) v1 from osd.2 2013-04-03 10:45:38.277255 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:45:38.327438 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 encode_pending v 29114 2013-04-03 10:45:38.361067 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 update_from_paxos applying incremental 29114 2013-04-03 10:45:38.361089 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 v29114: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 3513B/s wr, 0op/s 2013-04-03 10:45:38.361108 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 send_pg_creates to 0 pgs 2013-04-03 10:45:38.361113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 update_logger 2013-04-03 10:45:38.361124 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 create_pending v 29115 2013-04-03 10:45:38.361133 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:38.361156 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 check_osd_map already seen 605 >= 605 2013-04-03 10:45:38.361159 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 update_logger 2013-04-03 10:45:38.361170 7fcdffcdb700 0 log [INF] : pgmap v29114: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 3513B/s wr, 0op/s 2013-04-03 10:45:38.361236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos 2013-04-03 10:45:38.361259 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394105 2013-04-03 10:45:38.361279 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:38.361323 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos 2013-04-03 10:45:38.361338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394105 2013-04-03 10:45:38.361347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:38.361360 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:45:38.361394 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:38.361408 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:45:38.361419 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 logging 2013-04-03 10:45:37.235969 mon.0 10.252.24.25:6789/0 179 : [INF] pgmap v29113: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:45:38.361441 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 logging 2013-04-03 10:45:38.361171 mon.0 10.252.24.25:6789/0 180 : [INF] pgmap v29114: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 3513B/s wr, 0op/s 2013-04-03 10:45:38.375435 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 check_down_pgs 2013-04-03 10:45:38.375482 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 v29114: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 3513B/s wr, 0op/s 2013-04-03 10:45:38.375517 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:45:38.375548 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:45:38.375589 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:45:38.375621 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos 2013-04-03 10:45:38.375641 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394105 2013-04-03 10:45:38.375651 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 log 2013-04-03 10:45:38.375691 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:38.375712 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:45:38.385537 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos 2013-04-03 10:45:38.385558 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394105 2013-04-03 10:45:38.385576 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:39.360786 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 encode_pending v394106 2013-04-03 10:45:39.386984 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos 2013-04-03 10:45:39.387029 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos version 394106 summary v 394105 2013-04-03 10:45:39.387044 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos latest full 393798 2013-04-03 10:45:39.387063 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos applying incremental log 394106 2013-04-03 10:45:37.235969 mon.0 10.252.24.25:6789/0 179 : [INF] pgmap v29113: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:45:39.387090 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos applying incremental log 394106 2013-04-03 10:45:38.361171 mon.0 10.252.24.25:6789/0 180 : [INF] pgmap v29114: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 3513B/s wr, 0op/s 2013-04-03 10:45:39.387136 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 check_subs 2013-04-03 10:45:39.387168 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 create_pending v 394107 2013-04-03 10:45:39.387191 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:45:39.387258 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos 2013-04-03 10:45:39.387275 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos version 394106 summary v 394106 2013-04-03 10:45:39.387293 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:39.400304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos 2013-04-03 10:45:39.400325 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos version 394106 summary v 394106 2013-04-03 10:45:39.400343 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:39.597334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140468 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:40.943952 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:40.943963 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a51dc0 from 10.252.24.88:6789/0 2013-04-03 10:45:40.944018 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 183 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:45:41.177293 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33169/0) 2013-04-03 10:45:41.395341 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21839 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:45:41.631283 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:41.631294 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:45:41.631349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 preprocess_query pg_stats(1 pgs tid 56 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:41.631374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 prepare_update pg_stats(1 pgs tid 56 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:41.631380 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 prepare_pg_stats pg_stats(1 pgs tid 56 v 605) v1 from osd.1 2013-04-03 10:45:41.631387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:45:41.681562 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 encode_pending v 29115 2013-04-03 10:45:41.732797 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 update_from_paxos applying incremental 29115 2013-04-03 10:45:41.732820 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 v29115: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2726B/s wr, 0op/s 2013-04-03 10:45:41.732842 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 send_pg_creates to 0 pgs 2013-04-03 10:45:41.732846 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 update_logger 2013-04-03 10:45:41.732857 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 create_pending v 29116 2013-04-03 10:45:41.732867 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:41.732885 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 check_osd_map already seen 605 >= 605 2013-04-03 10:45:41.732888 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 update_logger 2013-04-03 10:45:41.732900 7fcdffcdb700 0 log [INF] : pgmap v29115: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2726B/s wr, 0op/s 2013-04-03 10:45:41.732973 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos 2013-04-03 10:45:41.732993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos version 394106 summary v 394106 2013-04-03 10:45:41.733013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:41.733057 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos 2013-04-03 10:45:41.733072 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos version 394106 summary v 394106 2013-04-03 10:45:41.733082 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:41.733094 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:45:41.733120 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:41.733133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:45:41.733146 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 logging 2013-04-03 10:45:41.732901 mon.0 10.252.24.25:6789/0 181 : [INF] pgmap v29115: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2726B/s wr, 0op/s 2013-04-03 10:45:41.758595 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos 2013-04-03 10:45:41.758617 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos version 394106 summary v 394106 2013-04-03 10:45:41.758637 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:41.797521 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:41.797533 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 2013-04-03 10:45:41.797589 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 preprocess_query pg_stats(1 pgs tid 30460 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:45:41.797614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 prepare_update pg_stats(1 pgs tid 30460 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:45:41.797619 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 prepare_pg_stats pg_stats(1 pgs tid 30460 v 605) v1 from osd.3 2013-04-03 10:45:41.797625 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:45:42.732569 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 encode_pending v394107 2013-04-03 10:45:42.733925 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 encode_pending v 29116 2013-04-03 10:45:42.760004 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos 2013-04-03 10:45:42.760044 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos version 394107 summary v 394106 2013-04-03 10:45:42.760059 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos latest full 393798 2013-04-03 10:45:42.760078 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos applying incremental log 394107 2013-04-03 10:45:41.732901 mon.0 10.252.24.25:6789/0 181 : [INF] pgmap v29115: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2726B/s wr, 0op/s 2013-04-03 10:45:42.760131 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 check_subs 2013-04-03 10:45:42.760163 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 create_pending v 394108 2013-04-03 10:45:42.760187 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:45:42.788828 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 update_from_paxos applying incremental 29116 2013-04-03 10:45:42.788850 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 v29116: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s 2013-04-03 10:45:42.788869 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 send_pg_creates to 0 pgs 2013-04-03 10:45:42.788874 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 update_logger 2013-04-03 10:45:42.788885 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 create_pending v 29117 2013-04-03 10:45:42.788895 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:45:42.788913 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 check_osd_map already seen 605 >= 605 2013-04-03 10:45:42.788916 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 update_logger 2013-04-03 10:45:42.788927 7fcdffcdb700 0 log [INF] : pgmap v29116: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s 2013-04-03 10:45:42.788993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos 2013-04-03 10:45:42.789013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos version 394107 summary v 394107 2013-04-03 10:45:42.789032 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:42.789076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos 2013-04-03 10:45:42.789091 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos version 394107 summary v 394107 2013-04-03 10:45:42.789101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:42.789112 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:45:42.789135 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:42.789147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:45:42.789158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 logging 2013-04-03 10:45:42.788928 mon.0 10.252.24.25:6789/0 182 : [INF] pgmap v29116: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s 2013-04-03 10:45:42.837928 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos 2013-04-03 10:45:42.837949 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos version 394107 summary v 394107 2013-04-03 10:45:42.837967 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:43.277854 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:43.277866 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:45:43.277919 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 preprocess_query pg_stats(1 pgs tid 39 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:43.277945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 prepare_update pg_stats(1 pgs tid 39 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:43.277952 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 prepare_pg_stats pg_stats(1 pgs tid 39 v 605) v1 from osd.2 2013-04-03 10:45:43.277959 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:45:43.375846 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 check_down_pgs 2013-04-03 10:45:43.375881 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 v29116: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s 2013-04-03 10:45:43.375913 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:45:43.375941 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:45:43.375979 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:45:43.376010 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos 2013-04-03 10:45:43.376030 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos version 394107 summary v 394107 2013-04-03 10:45:43.376040 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 log 2013-04-03 10:45:43.376057 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:43.376076 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:45:43.597452 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140469 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:43.785913 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 encode_pending v394108 2013-04-03 10:45:43.787360 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 encode_pending v 29117 2013-04-03 10:45:43.818266 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos 2013-04-03 10:45:43.818305 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos version 394108 summary v 394107 2013-04-03 10:45:43.818320 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos latest full 393798 2013-04-03 10:45:43.818342 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos applying incremental log 394108 2013-04-03 10:45:42.788928 mon.0 10.252.24.25:6789/0 182 : [INF] pgmap v29116: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s 2013-04-03 10:45:43.818393 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 check_subs 2013-04-03 10:45:43.818426 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 create_pending v 394109 2013-04-03 10:45:43.818449 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:45:43.843499 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 update_from_paxos applying incremental 29117 2013-04-03 10:45:43.843521 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 v29117: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2431B/s wr, 0op/s 2013-04-03 10:45:43.843540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 send_pg_creates to 0 pgs 2013-04-03 10:45:43.843545 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 update_logger 2013-04-03 10:45:43.843556 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 create_pending v 29118 2013-04-03 10:45:43.843566 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:43.843583 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 check_osd_map already seen 605 >= 605 2013-04-03 10:45:43.843587 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 update_logger 2013-04-03 10:45:43.843598 7fcdffcdb700 0 log [INF] : pgmap v29117: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2431B/s wr, 0op/s 2013-04-03 10:45:43.843672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos 2013-04-03 10:45:43.843692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos version 394108 summary v 394108 2013-04-03 10:45:43.843711 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:43.843757 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos 2013-04-03 10:45:43.843772 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos version 394108 summary v 394108 2013-04-03 10:45:43.843782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:43.843794 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:45:43.843819 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:43.843833 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:45:43.843844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 logging 2013-04-03 10:45:43.843599 mon.0 10.252.24.25:6789/0 183 : [INF] pgmap v29117: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2431B/s wr, 0op/s 2013-04-03 10:45:43.868451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos 2013-04-03 10:45:43.868472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos version 394108 summary v 394108 2013-04-03 10:45:43.868490 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:44.843156 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 encode_pending v394109 2013-04-03 10:45:44.868117 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos 2013-04-03 10:45:44.868156 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos version 394109 summary v 394108 2013-04-03 10:45:44.868171 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos latest full 393798 2013-04-03 10:45:44.868190 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos applying incremental log 394109 2013-04-03 10:45:43.843599 mon.0 10.252.24.25:6789/0 183 : [INF] pgmap v29117: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2431B/s wr, 0op/s 2013-04-03 10:45:44.868244 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 check_subs 2013-04-03 10:45:44.868277 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 create_pending v 394110 2013-04-03 10:45:44.868300 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:45:44.868366 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos 2013-04-03 10:45:44.868383 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos version 394109 summary v 394109 2013-04-03 10:45:44.868402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:44.879839 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos 2013-04-03 10:45:44.879861 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos version 394109 summary v 394109 2013-04-03 10:45:44.879878 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:44.944106 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:44.944116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a52b80 from 10.252.24.88:6789/0 2013-04-03 10:45:44.944169 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 184 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:45:45.395442 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21840 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:45:46.631838 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:46.631849 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:45:46.631906 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 preprocess_query pg_stats(1 pgs tid 57 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:46.631932 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 prepare_update pg_stats(1 pgs tid 57 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:46.631938 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 prepare_pg_stats pg_stats(1 pgs tid 57 v 605) v1 from osd.1 2013-04-03 10:45:46.631945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:45:46.682139 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 encode_pending v 29118 2013-04-03 10:45:46.712676 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 update_from_paxos applying incremental 29118 2013-04-03 10:45:46.712701 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 v29118: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2074B/s wr, 0op/s 2013-04-03 10:45:46.712721 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 send_pg_creates to 0 pgs 2013-04-03 10:45:46.712725 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 update_logger 2013-04-03 10:45:46.712736 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 create_pending v 29119 2013-04-03 10:45:46.712746 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:46.712769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 check_osd_map already seen 605 >= 605 2013-04-03 10:45:46.712773 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 update_logger 2013-04-03 10:45:46.712783 7fcdffcdb700 0 log [INF] : pgmap v29118: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2074B/s wr, 0op/s 2013-04-03 10:45:46.712847 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos 2013-04-03 10:45:46.712867 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos version 394109 summary v 394109 2013-04-03 10:45:46.712886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:46.712930 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos 2013-04-03 10:45:46.712945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos version 394109 summary v 394109 2013-04-03 10:45:46.712956 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:46.712968 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:45:46.712992 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:46.713005 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:45:46.713015 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 logging 2013-04-03 10:45:46.712785 mon.0 10.252.24.25:6789/0 184 : [INF] pgmap v29118: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2074B/s wr, 0op/s 2013-04-03 10:45:46.724974 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos 2013-04-03 10:45:46.724993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos version 394109 summary v 394109 2013-04-03 10:45:46.725009 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:47.597603 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140470 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:47.711845 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 encode_pending v394110 2013-04-03 10:45:47.853316 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos 2013-04-03 10:45:47.853355 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394109 2013-04-03 10:45:47.853370 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos latest full 393798 2013-04-03 10:45:47.853389 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos applying incremental log 394110 2013-04-03 10:45:46.712785 mon.0 10.252.24.25:6789/0 184 : [INF] pgmap v29118: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2074B/s wr, 0op/s 2013-04-03 10:45:47.853443 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 check_subs 2013-04-03 10:45:47.853475 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 create_pending v 394111 2013-04-03 10:45:47.853498 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:45:47.853565 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos 2013-04-03 10:45:47.853581 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 2013-04-03 10:45:47.853600 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:47.865069 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos 2013-04-03 10:45:47.865089 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 2013-04-03 10:45:47.865107 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:48.376199 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 check_down_pgs 2013-04-03 10:45:48.376232 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 v29118: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2074B/s wr, 0op/s 2013-04-03 10:45:48.376262 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:45:48.376288 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:45:48.376326 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:45:48.376360 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos 2013-04-03 10:45:48.376379 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 2013-04-03 10:45:48.376389 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 log 2013-04-03 10:45:48.376406 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:48.376425 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:45:48.944181 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:48.944193 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a512c0 from 10.252.24.88:6789/0 2013-04-03 10:45:48.944249 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 185 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:45:49.395540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21841 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:45:50.854234 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos 2013-04-03 10:45:50.854255 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 2013-04-03 10:45:50.854273 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:51.193338 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2500 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33170/0) 2013-04-03 10:45:51.597712 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140471 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:52.944328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:52.944339 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a51840 from 10.252.24.88:6789/0 2013-04-03 10:45:52.944397 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 186 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:45:53.376550 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 v29118: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2074B/s wr, 0op/s 2013-04-03 10:45:53.376593 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:45:53.376617 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:45:53.376665 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:45:53.376699 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos 2013-04-03 10:45:53.376719 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 2013-04-03 10:45:53.376729 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 log 2013-04-03 10:45:53.376746 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:53.376765 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:45:53.395630 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21842 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:45:53.854299 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos 2013-04-03 10:45:53.854321 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 2013-04-03 10:45:53.854338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:55.597802 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140472 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:56.632994 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:56.633010 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 2013-04-03 10:45:56.633080 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 preprocess_query pg_stats(1 pgs tid 58 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:56.633107 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 prepare_update pg_stats(1 pgs tid 58 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:56.633112 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 prepare_pg_stats pg_stats(1 pgs tid 58 v 605) v1 from osd.1 2013-04-03 10:45:56.633121 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:45:56.683332 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 encode_pending v 29119 2013-04-03 10:45:56.726006 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 update_from_paxos applying incremental 29119 2013-04-03 10:45:56.726028 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 v29119: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 635B/s wr, 0op/s 2013-04-03 10:45:56.726049 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 send_pg_creates to 0 pgs 2013-04-03 10:45:56.726054 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 update_logger 2013-04-03 10:45:56.726065 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 create_pending v 29120 2013-04-03 10:45:56.726074 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:45:56.726091 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 check_osd_map already seen 605 >= 605 2013-04-03 10:45:56.726095 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 update_logger 2013-04-03 10:45:56.726106 7fcdffcdb700 0 log [INF] : pgmap v29119: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 635B/s wr, 0op/s 2013-04-03 10:45:56.726170 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos 2013-04-03 10:45:56.726190 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 2013-04-03 10:45:56.726209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:56.726256 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos 2013-04-03 10:45:56.726271 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 2013-04-03 10:45:56.726281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:56.726293 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:45:56.726318 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:56.726331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:45:56.726343 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 logging 2013-04-03 10:45:56.726107 mon.0 10.252.24.25:6789/0 185 : [INF] pgmap v29119: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 635B/s wr, 0op/s 2013-04-03 10:45:56.737631 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos 2013-04-03 10:45:56.737653 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 2013-04-03 10:45:56.737672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:56.944562 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:56.944572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc4b00 from 10.252.24.88:6789/0 2013-04-03 10:45:56.944627 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 187 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:45:57.395712 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21843 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:45:57.725765 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 encode_pending v394111 2013-04-03 10:45:57.754430 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos 2013-04-03 10:45:57.754468 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos version 394111 summary v 394110 2013-04-03 10:45:57.754483 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos latest full 393798 2013-04-03 10:45:57.754501 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos applying incremental log 394111 2013-04-03 10:45:56.726107 mon.0 10.252.24.25:6789/0 185 : [INF] pgmap v29119: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 635B/s wr, 0op/s 2013-04-03 10:45:57.754553 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 check_subs 2013-04-03 10:45:57.754586 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 create_pending v 394112 2013-04-03 10:45:57.754609 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:45:57.754676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos 2013-04-03 10:45:57.754692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos version 394111 summary v 394111 2013-04-03 10:45:57.754711 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:57.779455 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos 2013-04-03 10:45:57.779476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos version 394111 summary v 394111 2013-04-03 10:45:57.779492 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:58.279548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:45:58.279560 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 2013-04-03 10:45:58.279613 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 preprocess_query pg_stats(1 pgs tid 40 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:58.279638 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 prepare_update pg_stats(1 pgs tid 40 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:58.279645 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 prepare_pg_stats pg_stats(1 pgs tid 40 v 605) v1 from osd.2 2013-04-03 10:45:58.279652 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:45:58.376923 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 check_down_pgs 2013-04-03 10:45:58.376985 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 v29119: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 635B/s wr, 0op/s 2013-04-03 10:45:58.377022 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:45:58.377051 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:45:58.377090 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:45:58.377123 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos 2013-04-03 10:45:58.377143 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos version 394111 summary v 394111 2013-04-03 10:45:58.377153 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 log 2013-04-03 10:45:58.377171 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:58.377191 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:45:58.754236 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 encode_pending v 29120 2013-04-03 10:45:58.793765 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 update_from_paxos applying incremental 29120 2013-04-03 10:45:58.793788 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29120 v29120: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1696B/s wr, 0op/s 2013-04-03 10:45:58.793807 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29120 send_pg_creates to 0 pgs 2013-04-03 10:45:58.793818 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29120 update_logger 2013-04-03 10:45:58.793831 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29120 create_pending v 29121 2013-04-03 10:45:58.793845 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29120 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:45:58.793865 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29120 check_osd_map already seen 605 >= 605 2013-04-03 10:45:58.793869 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29120 update_logger 2013-04-03 10:45:58.793881 7fcdffcdb700 0 log [INF] : pgmap v29120: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1696B/s wr, 0op/s 2013-04-03 10:45:58.793947 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos 2013-04-03 10:45:58.793967 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos version 394111 summary v 394111 2013-04-03 10:45:58.793986 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:58.794030 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos 2013-04-03 10:45:58.794046 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos version 394111 summary v 394111 2013-04-03 10:45:58.794055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:58.794066 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:45:58.794090 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:45:58.794102 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:45:58.794113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 logging 2013-04-03 10:45:58.793882 mon.0 10.252.24.25:6789/0 186 : [INF] pgmap v29120: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1696B/s wr, 0op/s 2013-04-03 10:45:58.807387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos 2013-04-03 10:45:58.807407 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos version 394111 summary v 394111 2013-04-03 10:45:58.807424 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:45:58.871470 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) 2013-04-03 10:45:58.871496 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader 2013-04-03 10:45:59.597926 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140473 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:45:59.787251 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394111 encode_pending v394112 2013-04-03 10:45:59.832457 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos 2013-04-03 10:45:59.832493 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394111 2013-04-03 10:45:59.832508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos latest full 393798 2013-04-03 10:45:59.832526 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394112 update_from_paxos applying incremental log 394112 2013-04-03 10:45:58.793882 mon.0 10.252.24.25:6789/0 186 : [INF] pgmap v29120: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1696B/s wr, 0op/s 2013-04-03 10:45:59.832577 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 check_subs 2013-04-03 10:45:59.832612 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 create_pending v 394113 2013-04-03 10:45:59.832634 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394112 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:45:59.832701 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos 2013-04-03 10:45:59.832717 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 2013-04-03 10:45:59.832738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:45:59.857515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos 2013-04-03 10:45:59.857536 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 2013-04-03 10:45:59.857554 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:00.944640 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:00.944650 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4840 from 10.252.24.88:6789/0 2013-04-03 10:46:00.944705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 188 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:46:01.395908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21844 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:46:02.833244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos 2013-04-03 10:46:02.833264 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 2013-04-03 10:46:02.833284 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:03.377319 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29120 check_down_pgs 2013-04-03 10:46:03.377353 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29120 v29120: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1696B/s wr, 0op/s 2013-04-03 10:46:03.377382 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:46:03.377570 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:46:03.377612 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:46:03.377644 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos 2013-04-03 10:46:03.377663 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 2013-04-03 10:46:03.377672 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394112 log 2013-04-03 10:46:03.377695 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:03.377715 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:46:03.598051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140474 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:46:04.944754 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:04.944764 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc5340 from 10.252.24.88:6789/0 2013-04-03 10:46:04.944818 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 189 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:46:05.395994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21845 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:46:05.833428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos 2013-04-03 10:46:05.833449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 2013-04-03 10:46:05.833468 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:07.598196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140475 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:46:08.377845 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29120 v29120: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1696B/s wr, 0op/s 2013-04-03 10:46:08.377888 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:46:08.377913 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:46:08.377960 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:46:08.377992 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos 2013-04-03 10:46:08.378010 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 2013-04-03 10:46:08.378019 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394112 log 2013-04-03 10:46:08.378047 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:08.378067 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:46:08.833460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos 2013-04-03 10:46:08.833481 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 2013-04-03 10:46:08.833498 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:08.944971 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:08.944981 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc5b80 from 10.252.24.88:6789/0 2013-04-03 10:46:08.945034 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 190 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:46:09.396065 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21846 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:46:11.225333 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2fe4780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33172/0) 2013-04-03 10:46:11.598328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140476 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:46:11.634599 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:11.634607 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 2013-04-03 10:46:11.634644 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29120 preprocess_query pg_stats(1 pgs tid 59 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:11.634670 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29120 prepare_update pg_stats(1 pgs tid 59 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:11.634676 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29120 prepare_pg_stats pg_stats(1 pgs tid 59 v 605) v1 from osd.1 2013-04-03 10:46:11.634682 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29120 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:46:11.684864 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29120 encode_pending v 29121 2013-04-03 10:46:11.723382 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29120 update_from_paxos applying incremental 29121 2013-04-03 10:46:11.723402 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 v29121: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:46:11.723421 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 send_pg_creates to 0 pgs 2013-04-03 10:46:11.723423 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 update_logger 2013-04-03 10:46:11.723433 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 create_pending v 29122 2013-04-03 10:46:11.723442 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29121 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:11.723460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 check_osd_map already seen 605 >= 605 2013-04-03 10:46:11.723463 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 update_logger 2013-04-03 10:46:11.723476 7fcdffcdb700 0 log [INF] : pgmap v29121: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:46:11.723542 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos 2013-04-03 10:46:11.723561 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 2013-04-03 10:46:11.723579 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:11.723622 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos 2013-04-03 10:46:11.723637 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 2013-04-03 10:46:11.723646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:11.723658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:46:11.723682 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:11.723694 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:46:11.723704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 logging 2013-04-03 10:46:11.723477 mon.0 10.252.24.25:6789/0 187 : [INF] pgmap v29121: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:46:11.736324 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos 2013-04-03 10:46:11.736345 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 2013-04-03 10:46:11.736365 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:11.800924 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:11.800935 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:46:11.800989 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 preprocess_query pg_stats(1 pgs tid 30461 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:46:11.801014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 prepare_update pg_stats(1 pgs tid 30461 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:46:11.801019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 prepare_pg_stats pg_stats(1 pgs tid 30461 v 605) v1 from osd.3 2013-04-03 10:46:11.801026 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:46:12.723097 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394112 encode_pending v394113 2013-04-03 10:46:12.724202 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29121 encode_pending v 29122 2013-04-03 10:46:12.893754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos 2013-04-03 10:46:12.893791 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos version 394113 summary v 394112 2013-04-03 10:46:12.893805 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos latest full 393798 2013-04-03 10:46:12.893823 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394113 update_from_paxos applying incremental log 394113 2013-04-03 10:46:11.723477 mon.0 10.252.24.25:6789/0 187 : [INF] pgmap v29121: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:46:12.893876 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 check_subs 2013-04-03 10:46:12.893908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 create_pending v 394114 2013-04-03 10:46:12.893931 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394113 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:46:12.931594 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29121 update_from_paxos applying incremental 29122 2013-04-03 10:46:12.931615 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 v29122: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 366B/s wr, 0op/s 2013-04-03 10:46:12.931634 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 send_pg_creates to 0 pgs 2013-04-03 10:46:12.931637 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 update_logger 2013-04-03 10:46:12.931647 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 create_pending v 29123 2013-04-03 10:46:12.931660 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29122 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:46:12.931679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 check_osd_map already seen 605 >= 605 2013-04-03 10:46:12.931682 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 update_logger 2013-04-03 10:46:12.931692 7fcdffcdb700 0 log [INF] : pgmap v29122: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 366B/s wr, 0op/s 2013-04-03 10:46:12.931767 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos 2013-04-03 10:46:12.931787 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos version 394113 summary v 394113 2013-04-03 10:46:12.931806 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:12.931849 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos 2013-04-03 10:46:12.931863 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos version 394113 summary v 394113 2013-04-03 10:46:12.931872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:12.931884 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:46:12.931909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:12.931920 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:46:12.931931 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 logging 2013-04-03 10:46:12.931701 mon.0 10.252.24.25:6789/0 188 : [INF] pgmap v29122: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 366B/s wr, 0op/s 2013-04-03 10:46:12.943074 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos 2013-04-03 10:46:12.943090 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos version 394113 summary v 394113 2013-04-03 10:46:12.943105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:12.944766 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:12.944775 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83580 from 10.252.24.88:6789/0 2013-04-03 10:46:12.944812 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 191 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:46:13.281367 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:13.281377 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:46:13.281428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 preprocess_query pg_stats(1 pgs tid 41 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:13.281452 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 prepare_update pg_stats(1 pgs tid 41 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:13.281458 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 prepare_pg_stats pg_stats(1 pgs tid 41 v 605) v1 from osd.2 2013-04-03 10:46:13.281464 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:46:13.378190 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29122 check_down_pgs 2013-04-03 10:46:13.378222 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29122 v29122: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 366B/s wr, 0op/s 2013-04-03 10:46:13.378252 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:46:13.378281 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:46:13.378319 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:46:13.378354 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos 2013-04-03 10:46:13.378374 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos version 394113 summary v 394113 2013-04-03 10:46:13.378384 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394113 log 2013-04-03 10:46:13.378402 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:13.378422 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:46:13.396257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21847 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:46:13.931308 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29122 encode_pending v 29123 2013-04-03 10:46:13.932592 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394113 encode_pending v394114 2013-04-03 10:46:14.051943 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29122 update_from_paxos applying incremental 29123 2013-04-03 10:46:14.051968 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 v29123: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 5925B/s wr, 0op/s 2013-04-03 10:46:14.051987 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 send_pg_creates to 0 pgs 2013-04-03 10:46:14.051990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 update_logger 2013-04-03 10:46:14.052000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 create_pending v 29124 2013-04-03 10:46:14.052009 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29123 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:14.052027 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 check_osd_map already seen 605 >= 605 2013-04-03 10:46:14.052030 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 update_logger 2013-04-03 10:46:14.052040 7fcdffcdb700 0 log [INF] : pgmap v29123: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 5925B/s wr, 0op/s 2013-04-03 10:46:14.089107 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos 2013-04-03 10:46:14.089144 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394113 2013-04-03 10:46:14.089159 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos latest full 393798 2013-04-03 10:46:14.089177 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394114 update_from_paxos applying incremental log 394114 2013-04-03 10:46:12.931701 mon.0 10.252.24.25:6789/0 188 : [INF] pgmap v29122: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 366B/s wr, 0op/s 2013-04-03 10:46:14.089227 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 check_subs 2013-04-03 10:46:14.089259 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 create_pending v 394115 2013-04-03 10:46:14.089282 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394114 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:46:14.089324 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos 2013-04-03 10:46:14.089339 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 2013-04-03 10:46:14.089348 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:14.089360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:46:14.089378 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 nothing new 2013-04-03 10:46:14.089433 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos 2013-04-03 10:46:14.089449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 2013-04-03 10:46:14.089467 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:14.101412 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos 2013-04-03 10:46:14.101429 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 2013-04-03 10:46:14.101444 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:15.598414 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140477 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:46:16.945212 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:16.945222 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a832c0 from 10.252.24.88:6789/0 2013-04-03 10:46:16.945278 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 192 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:46:17.089954 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos 2013-04-03 10:46:17.089975 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 2013-04-03 10:46:17.089993 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:17.396370 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21848 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:46:17.421121 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 2013-04-03 10:46:17.421132 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:46:18.282001 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:18.282011 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:46:18.282064 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 preprocess_query pg_stats(1 pgs tid 42 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:18.282088 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 prepare_update pg_stats(1 pgs tid 42 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:18.282094 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 prepare_pg_stats pg_stats(1 pgs tid 42 v 605) v1 from osd.2 2013-04-03 10:46:18.282100 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:46:18.332284 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29123 encode_pending v 29124 2013-04-03 10:46:18.378553 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:46:18.378593 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:46:18.378622 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos 2013-04-03 10:46:18.378641 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 2013-04-03 10:46:18.378650 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394114 log 2013-04-03 10:46:18.378668 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:18.378687 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:46:18.465397 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29123 update_from_paxos applying incremental 29124 2013-04-03 10:46:18.465419 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 v29124: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2921B/s wr, 0op/s 2013-04-03 10:46:18.465437 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 send_pg_creates to 0 pgs 2013-04-03 10:46:18.465440 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 update_logger 2013-04-03 10:46:18.465453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 create_pending v 29125 2013-04-03 10:46:18.465464 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29124 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:18.465482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 check_osd_map already seen 605 >= 605 2013-04-03 10:46:18.465485 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 update_logger 2013-04-03 10:46:18.465495 7fcdffcdb700 0 log [INF] : pgmap v29124: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2921B/s wr, 0op/s 2013-04-03 10:46:18.465564 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos 2013-04-03 10:46:18.465583 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 2013-04-03 10:46:18.465601 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:18.465646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos 2013-04-03 10:46:18.465661 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 2013-04-03 10:46:18.465670 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:18.465681 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:46:18.465714 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:18.465727 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:46:18.465737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 logging 2013-04-03 10:46:14.052042 mon.0 10.252.24.25:6789/0 189 : [INF] pgmap v29123: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 5925B/s wr, 0op/s 2013-04-03 10:46:18.465758 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 logging 2013-04-03 10:46:18.465496 mon.0 10.252.24.25:6789/0 190 : [INF] pgmap v29124: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2921B/s wr, 0op/s 2013-04-03 10:46:18.477010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos 2013-04-03 10:46:18.477030 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 2013-04-03 10:46:18.477047 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:19.465136 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394114 encode_pending v394115 2013-04-03 10:46:19.493775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos 2013-04-03 10:46:19.493827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos version 394115 summary v 394114 2013-04-03 10:46:19.493842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos latest full 393798 2013-04-03 10:46:19.493860 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394115 update_from_paxos applying incremental log 394115 2013-04-03 10:46:14.052042 mon.0 10.252.24.25:6789/0 189 : [INF] pgmap v29123: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 5925B/s wr, 0op/s 2013-04-03 10:46:19.493889 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394115 update_from_paxos applying incremental log 394115 2013-04-03 10:46:18.465496 mon.0 10.252.24.25:6789/0 190 : [INF] pgmap v29124: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2921B/s wr, 0op/s 2013-04-03 10:46:19.493947 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 check_subs 2013-04-03 10:46:19.493981 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 create_pending v 394116 2013-04-03 10:46:19.494003 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394115 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:46:19.494071 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos 2013-04-03 10:46:19.494086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos version 394115 summary v 394115 2013-04-03 10:46:19.494104 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:19.505433 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos 2013-04-03 10:46:19.505454 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos version 394115 summary v 394115 2013-04-03 10:46:19.505472 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:19.598526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140478 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:46:20.945247 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:20.945258 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83840 from 10.252.24.88:6789/0 2013-04-03 10:46:20.945316 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 193 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:46:21.241264 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2fe4c80 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33173/0) 2013-04-03 10:46:21.396489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21849 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:46:21.635938 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:21.635947 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:46:21.636000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 preprocess_query pg_stats(1 pgs tid 60 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:21.636025 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 prepare_update pg_stats(1 pgs tid 60 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:21.636031 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 prepare_pg_stats pg_stats(1 pgs tid 60 v 605) v1 from osd.1 2013-04-03 10:46:21.636037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:46:21.686238 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29124 encode_pending v 29125 2013-04-03 10:46:21.721189 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29124 update_from_paxos applying incremental 29125 2013-04-03 10:46:21.721211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 v29125: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1056B/s wr, 0op/s 2013-04-03 10:46:21.721229 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 send_pg_creates to 0 pgs 2013-04-03 10:46:21.721234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 update_logger 2013-04-03 10:46:21.721245 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 create_pending v 29126 2013-04-03 10:46:21.721257 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29125 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:21.721275 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 check_osd_map already seen 605 >= 605 2013-04-03 10:46:21.721277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 update_logger 2013-04-03 10:46:21.721287 7fcdffcdb700 0 log [INF] : pgmap v29125: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1056B/s wr, 0op/s 2013-04-03 10:46:21.721360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos 2013-04-03 10:46:21.721380 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos version 394115 summary v 394115 2013-04-03 10:46:21.721398 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:21.721444 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos 2013-04-03 10:46:21.721459 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos version 394115 summary v 394115 2013-04-03 10:46:21.721470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:21.721480 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:46:21.721505 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:21.721517 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:46:21.721527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 logging 2013-04-03 10:46:21.721288 mon.0 10.252.24.25:6789/0 191 : [INF] pgmap v29125: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1056B/s wr, 0op/s 2013-04-03 10:46:21.745719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos 2013-04-03 10:46:21.745739 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos version 394115 summary v 394115 2013-04-03 10:46:21.745756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:22.720425 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394115 encode_pending v394116 2013-04-03 10:46:22.778902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos 2013-04-03 10:46:22.778939 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394115 2013-04-03 10:46:22.778954 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos latest full 393798 2013-04-03 10:46:22.778987 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394116 update_from_paxos applying incremental log 394116 2013-04-03 10:46:21.721288 mon.0 10.252.24.25:6789/0 191 : [INF] pgmap v29125: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1056B/s wr, 0op/s 2013-04-03 10:46:22.779036 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 check_subs 2013-04-03 10:46:22.779069 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 create_pending v 394117 2013-04-03 10:46:22.779092 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394116 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:46:22.779159 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos 2013-04-03 10:46:22.779174 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394116 2013-04-03 10:46:22.779193 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:22.795306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos 2013-04-03 10:46:22.795323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394116 2013-04-03 10:46:22.795338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:23.378816 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29125 check_down_pgs 2013-04-03 10:46:23.378850 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29125 v29125: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1056B/s wr, 0op/s 2013-04-03 10:46:23.378885 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:46:23.378913 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:46:23.378950 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:46:23.378981 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos 2013-04-03 10:46:23.378999 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394116 2013-04-03 10:46:23.379008 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394116 log 2013-04-03 10:46:23.379025 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:23.379044 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:46:23.598709 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140479 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:46:24.945409 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:24.945419 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4dc0 from 10.252.24.88:6789/0 2013-04-03 10:46:24.945478 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 194 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:46:25.396547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21850 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:46:25.486874 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 2013-04-03 10:46:25.486886 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:46:25.779766 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos 2013-04-03 10:46:25.779786 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394116 2013-04-03 10:46:25.779804 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:26.636497 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:26.636507 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:46:26.636561 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 preprocess_query pg_stats(1 pgs tid 61 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:26.636586 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 prepare_update pg_stats(1 pgs tid 61 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:26.636592 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 prepare_pg_stats pg_stats(1 pgs tid 61 v 605) v1 from osd.1 2013-04-03 10:46:26.636598 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:46:26.686780 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29125 encode_pending v 29126 2013-04-03 10:46:26.734619 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29125 update_from_paxos applying incremental 29126 2013-04-03 10:46:26.734640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 v29126: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:46:26.734659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 send_pg_creates to 0 pgs 2013-04-03 10:46:26.734663 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 update_logger 2013-04-03 10:46:26.734674 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 create_pending v 29127 2013-04-03 10:46:26.734683 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29126 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:26.734700 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 check_osd_map already seen 605 >= 605 2013-04-03 10:46:26.734703 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 update_logger 2013-04-03 10:46:26.734714 7fcdffcdb700 0 log [INF] : pgmap v29126: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:46:26.734776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos 2013-04-03 10:46:26.734795 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394116 2013-04-03 10:46:26.734814 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:26.734860 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos 2013-04-03 10:46:26.734874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394116 2013-04-03 10:46:26.734883 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:26.734896 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:46:26.734919 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:26.734931 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:46:26.734941 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 logging 2013-04-03 10:46:26.734715 mon.0 10.252.24.25:6789/0 192 : [INF] pgmap v29126: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:46:26.745941 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos 2013-04-03 10:46:26.745959 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394116 2013-04-03 10:46:26.745974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:27.598822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140480 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:46:27.734356 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394116 encode_pending v394117 2013-04-03 10:46:27.761725 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos 2013-04-03 10:46:27.761762 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos version 394117 summary v 394116 2013-04-03 10:46:27.761776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos latest full 393798 2013-04-03 10:46:27.761794 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394117 update_from_paxos applying incremental log 394117 2013-04-03 10:46:26.734715 mon.0 10.252.24.25:6789/0 192 : [INF] pgmap v29126: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:46:27.761854 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 check_subs 2013-04-03 10:46:27.761888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 create_pending v 394118 2013-04-03 10:46:27.761910 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394117 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:46:27.761976 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos 2013-04-03 10:46:27.761992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos version 394117 summary v 394117 2013-04-03 10:46:27.762010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:27.773098 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos 2013-04-03 10:46:27.773115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos version 394117 summary v 394117 2013-04-03 10:46:27.773524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:28.283252 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:28.283265 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 2013-04-03 10:46:28.283318 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 preprocess_query pg_stats(1 pgs tid 43 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:28.283343 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 prepare_update pg_stats(1 pgs tid 43 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:28.283348 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 prepare_pg_stats pg_stats(1 pgs tid 43 v 605) v1 from osd.2 2013-04-03 10:46:28.283355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:46:28.379176 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29126 check_down_pgs 2013-04-03 10:46:28.379207 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29126 v29126: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:46:28.379239 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:46:28.379264 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:46:28.379301 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:46:28.379703 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos 2013-04-03 10:46:28.379725 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos version 394117 summary v 394117 2013-04-03 10:46:28.379734 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394117 log 2013-04-03 10:46:28.379751 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:28.379772 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:46:28.761500 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29126 encode_pending v 29127 2013-04-03 10:46:28.811683 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29126 update_from_paxos applying incremental 29127 2013-04-03 10:46:28.811706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29127 v29127: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1736B/s wr, 0op/s 2013-04-03 10:46:28.811726 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29127 send_pg_creates to 0 pgs 2013-04-03 10:46:28.811729 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29127 update_logger 2013-04-03 10:46:28.811739 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29127 create_pending v 29128 2013-04-03 10:46:28.811748 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29127 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:28.811765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29127 check_osd_map already seen 605 >= 605 2013-04-03 10:46:28.811767 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29127 update_logger 2013-04-03 10:46:28.811777 7fcdffcdb700 0 log [INF] : pgmap v29127: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1736B/s wr, 0op/s 2013-04-03 10:46:28.811850 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos 2013-04-03 10:46:28.811869 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos version 394117 summary v 394117 2013-04-03 10:46:28.811889 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:28.811934 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos 2013-04-03 10:46:28.811948 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos version 394117 summary v 394117 2013-04-03 10:46:28.811957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:28.811969 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:46:28.811994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:28.812006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:46:28.812018 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 logging 2013-04-03 10:46:28.811778 mon.0 10.252.24.25:6789/0 193 : [INF] pgmap v29127: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1736B/s wr, 0op/s 2013-04-03 10:46:28.824306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos 2013-04-03 10:46:28.824323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos version 394117 summary v 394117 2013-04-03 10:46:28.824339 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:28.927618 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:46:28.927625 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:46:28.927634 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 2013-04-03 10:46:28.945441 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:28.945451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc4b00 from 10.252.24.88:6789/0 2013-04-03 10:46:28.945506 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 195 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:46:29.396698 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21851 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:46:29.811472 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394117 encode_pending v394118 2013-04-03 10:46:29.871888 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos 2013-04-03 10:46:29.871933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394117 2013-04-03 10:46:29.871948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos latest full 393798 2013-04-03 10:46:29.872624 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos applying incremental log 394118 2013-04-03 10:46:28.811778 mon.0 10.252.24.25:6789/0 193 : [INF] pgmap v29127: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1736B/s wr, 0op/s 2013-04-03 10:46:29.872670 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 check_subs 2013-04-03 10:46:29.872713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 create_pending v 394119 2013-04-03 10:46:29.872739 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:46:29.872810 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos 2013-04-03 10:46:29.872827 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 2013-04-03 10:46:29.872846 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:29.896563 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos 2013-04-03 10:46:29.896584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 2013-04-03 10:46:29.896659 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:31.257248 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2fe4000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33175/0) 2013-04-03 10:46:31.599010 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140481 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:46:32.872786 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos 2013-04-03 10:46:32.872807 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 2013-04-03 10:46:32.872826 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:32.945619 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:32.945631 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc5080 from 10.252.24.88:6789/0 2013-04-03 10:46:32.945684 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 196 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:46:33.379900 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 check_down_pgs 2013-04-03 10:46:33.379934 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 v29127: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1736B/s wr, 0op/s 2013-04-03 10:46:33.379964 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:46:33.379990 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:46:33.380028 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:46:33.380060 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos 2013-04-03 10:46:33.380080 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 2013-04-03 10:46:33.380090 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 log 2013-04-03 10:46:33.380107 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:33.380125 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:46:33.396814 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21852 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:46:35.599076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140482 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:46:35.872842 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos 2013-04-03 10:46:35.872863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 2013-04-03 10:46:35.872881 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:36.637992 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:36.638004 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:46:36.638061 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 preprocess_query pg_stats(1 pgs tid 62 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:36.638086 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 prepare_update pg_stats(1 pgs tid 62 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:36.638093 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 prepare_pg_stats pg_stats(1 pgs tid 62 v 605) v1 from osd.1 2013-04-03 10:46:36.638100 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:46:36.688283 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 encode_pending v 29128 2013-04-03 10:46:36.725961 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 update_from_paxos applying incremental 29128 2013-04-03 10:46:36.725983 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 v29128: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s 2013-04-03 10:46:36.726003 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 send_pg_creates to 0 pgs 2013-04-03 10:46:36.726006 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 update_logger 2013-04-03 10:46:36.726017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 create_pending v 29129 2013-04-03 10:46:36.726033 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:36.726051 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 check_osd_map already seen 605 >= 605 2013-04-03 10:46:36.726054 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 update_logger 2013-04-03 10:46:36.726065 7fcdffcdb700 0 log [INF] : pgmap v29128: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s 2013-04-03 10:46:36.726128 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos 2013-04-03 10:46:36.726148 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 2013-04-03 10:46:36.726167 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:36.726209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos 2013-04-03 10:46:36.726224 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 2013-04-03 10:46:36.726233 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:36.726248 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:46:36.726275 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:36.726288 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:46:36.726301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 logging 2013-04-03 10:46:36.726066 mon.0 10.252.24.25:6789/0 194 : [INF] pgmap v29128: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s 2013-04-03 10:46:36.734873 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos 2013-04-03 10:46:36.734893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 2013-04-03 10:46:36.734909 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:36.803449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:36.803463 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 2013-04-03 10:46:36.803516 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 preprocess_query pg_stats(1 pgs tid 30462 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:46:36.803542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 prepare_update pg_stats(1 pgs tid 30462 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:46:36.803548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 prepare_pg_stats pg_stats(1 pgs tid 30462 v 605) v1 from osd.3 2013-04-03 10:46:36.803555 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:46:36.945801 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:36.945812 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83580 from 10.252.24.88:6789/0 2013-04-03 10:46:36.945865 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 197 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:46:37.396914 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21853 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:46:37.715024 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 encode_pending v 29129 2013-04-03 10:46:37.716509 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 encode_pending v394119 2013-04-03 10:46:37.742576 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 update_from_paxos applying incremental 29129 2013-04-03 10:46:37.742598 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 v29129: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 571B/s wr, 0op/s 2013-04-03 10:46:37.742617 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 send_pg_creates to 0 pgs 2013-04-03 10:46:37.742620 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 update_logger 2013-04-03 10:46:37.742631 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 create_pending v 29130 2013-04-03 10:46:37.742640 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:46:37.742657 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 check_osd_map already seen 605 >= 605 2013-04-03 10:46:37.742661 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 update_logger 2013-04-03 10:46:37.742672 7fcdffcdb700 0 log [INF] : pgmap v29129: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 571B/s wr, 0op/s 2013-04-03 10:46:37.779768 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos 2013-04-03 10:46:37.779806 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394118 2013-04-03 10:46:37.779820 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos latest full 393798 2013-04-03 10:46:37.779842 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos applying incremental log 394119 2013-04-03 10:46:36.726066 mon.0 10.252.24.25:6789/0 194 : [INF] pgmap v29128: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s 2013-04-03 10:46:37.779894 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 check_subs 2013-04-03 10:46:37.779926 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 create_pending v 394120 2013-04-03 10:46:37.779949 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:46:37.779983 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos 2013-04-03 10:46:37.779998 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 2013-04-03 10:46:37.780008 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:37.780021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:46:37.780040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 nothing new 2013-04-03 10:46:37.780096 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos 2013-04-03 10:46:37.780112 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 2013-04-03 10:46:37.780132 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:37.791706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos 2013-04-03 10:46:37.791734 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 2013-04-03 10:46:37.791752 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:38.380251 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 check_down_pgs 2013-04-03 10:46:38.380286 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 v29129: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 571B/s wr, 0op/s 2013-04-03 10:46:38.380316 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:46:38.380341 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:46:38.380376 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:46:38.380409 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos 2013-04-03 10:46:38.380429 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 2013-04-03 10:46:38.380439 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 log 2013-04-03 10:46:38.380456 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:38.380476 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:46:39.599168 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140483 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:46:40.780675 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos 2013-04-03 10:46:40.780696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 2013-04-03 10:46:40.780714 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:40.945912 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:40.945925 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273a580 from 10.252.24.88:6789/0 2013-04-03 10:46:40.945979 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 198 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:46:41.273288 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x300c280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33176/0) 2013-04-03 10:46:41.397048 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21854 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:46:43.285153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:43.285165 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 2013-04-03 10:46:43.285216 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 preprocess_query pg_stats(1 pgs tid 44 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:43.285242 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 prepare_update pg_stats(1 pgs tid 44 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:43.285247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 prepare_pg_stats pg_stats(1 pgs tid 44 v 605) v1 from osd.2 2013-04-03 10:46:43.285253 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:46:43.335442 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 encode_pending v 29130 2013-04-03 10:46:43.367436 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 update_from_paxos applying incremental 29130 2013-04-03 10:46:43.367452 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 v29130: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2002B/s wr, 0op/s 2013-04-03 10:46:43.367472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 send_pg_creates to 0 pgs 2013-04-03 10:46:43.367479 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 update_logger 2013-04-03 10:46:43.367489 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 create_pending v 29131 2013-04-03 10:46:43.367498 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:43.367517 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 check_osd_map already seen 605 >= 605 2013-04-03 10:46:43.367521 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 update_logger 2013-04-03 10:46:43.367532 7fcdffcdb700 0 log [INF] : pgmap v29130: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2002B/s wr, 0op/s 2013-04-03 10:46:43.367596 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos 2013-04-03 10:46:43.367616 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 2013-04-03 10:46:43.367635 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:43.367678 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos 2013-04-03 10:46:43.367694 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 2013-04-03 10:46:43.367703 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:43.367718 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:46:43.367750 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:43.367763 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:46:43.367775 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 logging 2013-04-03 10:46:37.742673 mon.0 10.252.24.25:6789/0 195 : [INF] pgmap v29129: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 571B/s wr, 0op/s 2013-04-03 10:46:43.367797 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 logging 2013-04-03 10:46:43.367533 mon.0 10.252.24.25:6789/0 196 : [INF] pgmap v29130: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2002B/s wr, 0op/s 2013-04-03 10:46:43.380588 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 check_down_pgs 2013-04-03 10:46:43.380618 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 v29130: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2002B/s wr, 0op/s 2013-04-03 10:46:43.380648 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:46:43.380670 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:46:43.380710 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:46:43.380741 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos 2013-04-03 10:46:43.380759 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 2013-04-03 10:46:43.380768 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 log 2013-04-03 10:46:43.380784 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:43.380802 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:46:43.390006 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos 2013-04-03 10:46:43.390023 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 2013-04-03 10:46:43.390039 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:43.599361 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140484 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:46:44.365143 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 encode_pending v394120 2013-04-03 10:46:44.388951 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos 2013-04-03 10:46:44.388979 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394119 2013-04-03 10:46:44.388993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos latest full 393798 2013-04-03 10:46:44.389012 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos applying incremental log 394120 2013-04-03 10:46:37.742673 mon.0 10.252.24.25:6789/0 195 : [INF] pgmap v29129: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 571B/s wr, 0op/s 2013-04-03 10:46:44.389044 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos applying incremental log 394120 2013-04-03 10:46:43.367533 mon.0 10.252.24.25:6789/0 196 : [INF] pgmap v29130: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2002B/s wr, 0op/s 2013-04-03 10:46:44.389081 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 check_subs 2013-04-03 10:46:44.389111 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 create_pending v 394121 2013-04-03 10:46:44.389134 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:46:44.389199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos 2013-04-03 10:46:44.389216 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394120 2013-04-03 10:46:44.389235 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:44.400257 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos 2013-04-03 10:46:44.400274 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394120 2013-04-03 10:46:44.400290 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:44.946037 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:44.946048 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273b080 from 10.252.24.88:6789/0 2013-04-03 10:46:44.946104 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 199 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:46:45.397112 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21855 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:46:47.389755 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos 2013-04-03 10:46:47.389776 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394120 2013-04-03 10:46:47.389794 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:47.599465 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140485 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:46:48.286012 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:48.286024 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:46:48.286076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 preprocess_query pg_stats(1 pgs tid 45 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:48.286101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 prepare_update pg_stats(1 pgs tid 45 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:48.286107 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 prepare_pg_stats pg_stats(1 pgs tid 45 v 605) v1 from osd.2 2013-04-03 10:46:48.286114 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:46:48.336307 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 encode_pending v 29131 2013-04-03 10:46:48.363562 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 update_from_paxos applying incremental 29131 2013-04-03 10:46:48.363578 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 v29131: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1542B/s wr, 0op/s 2013-04-03 10:46:48.363598 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 send_pg_creates to 0 pgs 2013-04-03 10:46:48.363602 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 update_logger 2013-04-03 10:46:48.363613 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 create_pending v 29132 2013-04-03 10:46:48.363622 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:48.363639 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 check_osd_map already seen 605 >= 605 2013-04-03 10:46:48.363642 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 update_logger 2013-04-03 10:46:48.363653 7fcdffcdb700 0 log [INF] : pgmap v29131: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1542B/s wr, 0op/s 2013-04-03 10:46:48.363713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos 2013-04-03 10:46:48.363732 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394120 2013-04-03 10:46:48.363755 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:48.363798 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos 2013-04-03 10:46:48.363813 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394120 2013-04-03 10:46:48.363822 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:48.363833 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:46:48.363858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:48.363871 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:46:48.363881 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 logging 2013-04-03 10:46:48.363654 mon.0 10.252.24.25:6789/0 197 : [INF] pgmap v29131: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1542B/s wr, 0op/s 2013-04-03 10:46:48.380907 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 check_down_pgs 2013-04-03 10:46:48.380938 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 v29131: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1542B/s wr, 0op/s 2013-04-03 10:46:48.380966 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:46:48.380987 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:46:48.381024 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:46:48.381055 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos 2013-04-03 10:46:48.381072 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394120 2013-04-03 10:46:48.381081 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 log 2013-04-03 10:46:48.381097 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:48.381115 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:46:48.385970 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos 2013-04-03 10:46:48.385987 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394120 2013-04-03 10:46:48.386004 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:48.946104 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:48.946115 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2f7a600 from 10.252.24.88:6789/0 2013-04-03 10:46:48.946169 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 200 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:46:49.362680 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 encode_pending v394121 2013-04-03 10:46:49.393334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos 2013-04-03 10:46:49.393371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos version 394121 summary v 394120 2013-04-03 10:46:49.393386 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos latest full 393798 2013-04-03 10:46:49.393405 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos applying incremental log 394121 2013-04-03 10:46:48.363654 mon.0 10.252.24.25:6789/0 197 : [INF] pgmap v29131: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1542B/s wr, 0op/s 2013-04-03 10:46:49.393454 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 check_subs 2013-04-03 10:46:49.393486 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 create_pending v 394122 2013-04-03 10:46:49.393509 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:46:49.393579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos 2013-04-03 10:46:49.393595 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos version 394121 summary v 394121 2013-04-03 10:46:49.393614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:49.397172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21856 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:46:49.404701 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos 2013-04-03 10:46:49.404720 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos version 394121 summary v 394121 2013-04-03 10:46:49.404737 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:51.599617 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140486 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:46:51.639507 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:51.639516 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:46:51.639897 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 preprocess_query pg_stats(1 pgs tid 63 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:51.639927 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 prepare_update pg_stats(1 pgs tid 63 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:51.639933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 prepare_pg_stats pg_stats(1 pgs tid 63 v 605) v1 from osd.1 2013-04-03 10:46:51.639940 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:46:51.690131 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 encode_pending v 29132 2013-04-03 10:46:51.735665 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 update_from_paxos applying incremental 29132 2013-04-03 10:46:51.735685 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 v29132: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:46:51.735706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 send_pg_creates to 0 pgs 2013-04-03 10:46:51.735711 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 update_logger 2013-04-03 10:46:51.735722 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 create_pending v 29133 2013-04-03 10:46:51.735731 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:51.735748 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 check_osd_map already seen 605 >= 605 2013-04-03 10:46:51.735752 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 update_logger 2013-04-03 10:46:51.735763 7fcdffcdb700 0 log [INF] : pgmap v29132: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:46:51.735825 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos 2013-04-03 10:46:51.735845 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos version 394121 summary v 394121 2013-04-03 10:46:51.735864 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:51.735906 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos 2013-04-03 10:46:51.735921 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos version 394121 summary v 394121 2013-04-03 10:46:51.735931 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:51.735945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:46:51.735968 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:51.735980 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:46:51.735991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 logging 2013-04-03 10:46:51.735764 mon.0 10.252.24.25:6789/0 198 : [INF] pgmap v29132: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:46:51.760184 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos 2013-04-03 10:46:51.760204 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos version 394121 summary v 394121 2013-04-03 10:46:51.760221 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:52.735455 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 encode_pending v394122 2013-04-03 10:46:52.892576 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos 2013-04-03 10:46:52.892607 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394121 2013-04-03 10:46:52.892622 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos latest full 393798 2013-04-03 10:46:52.892643 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos applying incremental log 394122 2013-04-03 10:46:51.735764 mon.0 10.252.24.25:6789/0 198 : [INF] pgmap v29132: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:46:52.892689 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 check_subs 2013-04-03 10:46:52.892722 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 create_pending v 394123 2013-04-03 10:46:52.892745 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:46:52.892814 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos 2013-04-03 10:46:52.892831 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394122 2013-04-03 10:46:52.892850 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:52.916975 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos 2013-04-03 10:46:52.916992 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394122 2013-04-03 10:46:52.917008 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:52.946136 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:52.946147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2f7a080 from 10.252.24.88:6789/0 2013-04-03 10:46:52.946202 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 201 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:46:53.381237 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 check_down_pgs 2013-04-03 10:46:53.381265 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 v29132: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:46:53.381294 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:46:53.381326 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:46:53.381365 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:46:53.381397 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos 2013-04-03 10:46:53.381416 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394122 2013-04-03 10:46:53.381426 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 log 2013-04-03 10:46:53.381443 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:53.381463 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:46:53.397338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21857 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:46:55.599709 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140487 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:46:55.893419 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos 2013-04-03 10:46:55.893440 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394122 2013-04-03 10:46:55.893458 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:56.640142 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:56.640153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 2013-04-03 10:46:56.640205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 preprocess_query pg_stats(1 pgs tid 64 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:56.640231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 prepare_update pg_stats(1 pgs tid 64 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:56.640237 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 prepare_pg_stats pg_stats(1 pgs tid 64 v 605) v1 from osd.1 2013-04-03 10:46:56.640244 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:46:56.690413 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 encode_pending v 29133 2013-04-03 10:46:56.719628 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 update_from_paxos applying incremental 29133 2013-04-03 10:46:56.719646 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 v29133: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:46:56.719666 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 send_pg_creates to 0 pgs 2013-04-03 10:46:56.719669 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 update_logger 2013-04-03 10:46:56.719683 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 create_pending v 29134 2013-04-03 10:46:56.719692 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:46:56.719710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 check_osd_map already seen 605 >= 605 2013-04-03 10:46:56.719713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 update_logger 2013-04-03 10:46:56.719725 7fcdffcdb700 0 log [INF] : pgmap v29133: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:46:56.719788 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos 2013-04-03 10:46:56.719808 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394122 2013-04-03 10:46:56.719827 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:56.719870 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos 2013-04-03 10:46:56.719886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394122 2013-04-03 10:46:56.719895 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:56.719907 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:46:56.719933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:56.719948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:46:56.719960 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 logging 2013-04-03 10:46:56.719726 mon.0 10.252.24.25:6789/0 199 : [INF] pgmap v29133: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:46:56.744161 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos 2013-04-03 10:46:56.744180 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394122 2013-04-03 10:46:56.744196 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:56.805598 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:56.805607 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 2013-04-03 10:46:56.805638 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 preprocess_query pg_stats(1 pgs tid 30463 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:46:56.805659 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 prepare_update pg_stats(1 pgs tid 30463 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:46:56.805665 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 prepare_pg_stats pg_stats(1 pgs tid 30463 v 605) v1 from osd.3 2013-04-03 10:46:56.805672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:46:56.946316 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:56.946325 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2f79000 from 10.252.24.88:6789/0 2013-04-03 10:46:56.946359 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 202 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:46:57.397458 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21858 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:46:57.719414 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 encode_pending v 29134 2013-04-03 10:46:57.720672 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 encode_pending v394123 2013-04-03 10:46:57.876478 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 update_from_paxos applying incremental 29134 2013-04-03 10:46:57.876498 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 v29134: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 849B/s wr, 0op/s 2013-04-03 10:46:57.876519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 send_pg_creates to 0 pgs 2013-04-03 10:46:57.876523 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 update_logger 2013-04-03 10:46:57.876534 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 create_pending v 29135 2013-04-03 10:46:57.876544 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 _updated_stats for osd.3 10.252.24.88:6801/17538 2013-04-03 10:46:57.876561 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 check_osd_map already seen 605 >= 605 2013-04-03 10:46:57.876565 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 update_logger 2013-04-03 10:46:57.876575 7fcdffcdb700 0 log [INF] : pgmap v29134: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 849B/s wr, 0op/s 2013-04-03 10:46:57.901134 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos 2013-04-03 10:46:57.901158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos version 394123 summary v 394122 2013-04-03 10:46:57.901172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos latest full 393798 2013-04-03 10:46:57.901191 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos applying incremental log 394123 2013-04-03 10:46:56.719726 mon.0 10.252.24.25:6789/0 199 : [INF] pgmap v29133: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s 2013-04-03 10:46:57.901441 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 check_subs 2013-04-03 10:46:57.901474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 create_pending v 394124 2013-04-03 10:46:57.901497 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:46:57.901531 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos 2013-04-03 10:46:57.901547 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos version 394123 summary v 394123 2013-04-03 10:46:57.901556 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:57.901569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:46:57.901589 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 nothing new 2013-04-03 10:46:57.901637 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos 2013-04-03 10:46:57.901653 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos version 394123 summary v 394123 2013-04-03 10:46:57.901672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:57.912542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos 2013-04-03 10:46:57.912559 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos version 394123 summary v 394123 2013-04-03 10:46:57.912577 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:58.287130 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:46:58.287141 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:46:58.287191 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 preprocess_query pg_stats(1 pgs tid 46 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:58.287216 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 prepare_update pg_stats(1 pgs tid 46 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:58.287223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 prepare_pg_stats pg_stats(1 pgs tid 46 v 605) v1 from osd.2 2013-04-03 10:46:58.287229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:46:58.381587 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 check_down_pgs 2013-04-03 10:46:58.381619 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 v29134: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 849B/s wr, 0op/s 2013-04-03 10:46:58.381649 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:46:58.381672 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:46:58.381710 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 2013-04-03 10:46:58.381741 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos 2013-04-03 10:46:58.381761 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos version 394123 summary v 394123 2013-04-03 10:46:58.381773 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 log 2013-04-03 10:46:58.381791 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos 2013-04-03 10:46:58.381810 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth 2013-04-03 10:46:58.901001 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 encode_pending v 29135 2013-04-03 10:46:58.927688 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) 2013-04-03 10:46:58.927703 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader 2013-04-03 10:46:58.928390 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29134 update_from_paxos applying incremental 29135 2013-04-03 10:46:58.928410 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 v29135: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 33815B/s wr, 0op/s 2013-04-03 10:46:58.928430 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 send_pg_creates to 0 pgs 2013-04-03 10:46:58.928432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 update_logger 2013-04-03 10:46:58.928442 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 create_pending v 29136 2013-04-03 10:46:58.928451 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29135 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:46:58.928468 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 check_osd_map already seen 605 >= 605 2013-04-03 10:46:58.928471 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 update_logger 2013-04-03 10:46:58.928481 7fcdffcdb700 0 log [INF] : pgmap v29135: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 33815B/s wr, 0op/s 2013-04-03 10:46:58.928556 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 update_from_paxos 2013-04-03 10:46:58.928575 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 update_from_paxos version 394123 summary v 394123 2013-04-03 10:46:58.928595 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:58.928640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 update_from_paxos 2013-04-03 10:46:58.928655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 update_from_paxos version 394123 summary v 394123 2013-04-03 10:46:58.928664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:58.928676 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:46:58.928708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:46:58.928720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:46:58.928731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 logging 2013-04-03 10:46:57.876576 mon.0 10.252.24.25:6789/0 200 : [INF] pgmap v29134: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 849B/s wr, 0op/s 2013-04-03 10:46:58.928752 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 logging 2013-04-03 10:46:58.928482 mon.0 10.252.24.25:6789/0 201 : [INF] pgmap v29135: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 33815B/s wr, 0op/s 2013-04-03 10:46:58.939950 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 update_from_paxos 2013-04-03 10:46:58.939966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 update_from_paxos version 394123 summary v 394123 2013-04-03 10:46:58.939983 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:46:59.599849 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140488 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:46:59.928178 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394123 encode_pending v394124 2013-04-03 10:46:59.980333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos 2013-04-03 10:46:59.980360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394123 2013-04-03 10:46:59.980374 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos latest full 393798 2013-04-03 10:46:59.980394 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394124 update_from_paxos applying incremental log 394124 2013-04-03 10:46:57.876576 mon.0 10.252.24.25:6789/0 200 : [INF] pgmap v29134: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 849B/s wr, 0op/s 2013-04-03 10:46:59.980422 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394124 update_from_paxos applying incremental log 394124 2013-04-03 10:46:58.928482 mon.0 10.252.24.25:6789/0 201 : [INF] pgmap v29135: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 33815B/s wr, 0op/s 2013-04-03 10:46:59.980464 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 check_subs 2013-04-03 10:46:59.980496 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 create_pending v 394125 2013-04-03 10:46:59.980518 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394124 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:46:59.980582 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos 2013-04-03 10:46:59.980597 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 2013-04-03 10:46:59.980615 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:00.004897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos 2013-04-03 10:47:00.004919 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 2013-04-03 10:47:00.004935 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:00.946504 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:00.946514 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a512c0 from 10.252.24.88:6789/0 2013-04-03 10:47:00.946568 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 203 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:47:01.305226 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x300cc80 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33179/0) 2013-04-03 10:47:01.397556 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21859 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:47:02.981221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos 2013-04-03 10:47:02.981240 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 2013-04-03 10:47:02.981258 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:03.381928 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29135 check_down_pgs 2013-04-03 10:47:03.381958 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29135 v29135: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 33815B/s wr, 0op/s 2013-04-03 10:47:03.381986 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:03.382010 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:47:03.382047 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:47:03.382082 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos 2013-04-03 10:47:03.382100 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 2013-04-03 10:47:03.382109 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394124 log 2013-04-03 10:47:03.382126 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:03.382145 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:47:03.599977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140489 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:47:04.946692 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:04.946705 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52b80 from 10.252.24.88:6789/0 2013-04-03 10:47:04.946759 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 204 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:47:05.397656 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21860 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:47:05.981277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos 2013-04-03 10:47:05.981297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 2013-04-03 10:47:05.981318 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:06.641423 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:06.641432 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 2013-04-03 10:47:06.641485 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 preprocess_query pg_stats(1 pgs tid 65 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:06.641721 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 prepare_update pg_stats(1 pgs tid 65 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:06.641732 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 prepare_pg_stats pg_stats(1 pgs tid 65 v 605) v1 from osd.1 2013-04-03 10:47:06.641738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:47:06.691924 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29135 encode_pending v 29136 2013-04-03 10:47:06.719201 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29135 update_from_paxos applying incremental 29136 2013-04-03 10:47:06.719217 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 v29136: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 8673B/s wr, 0op/s 2013-04-03 10:47:06.719236 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 send_pg_creates to 0 pgs 2013-04-03 10:47:06.719239 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 update_logger 2013-04-03 10:47:06.719249 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 create_pending v 29137 2013-04-03 10:47:06.719258 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29136 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:06.719274 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 check_osd_map already seen 605 >= 605 2013-04-03 10:47:06.719277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 update_logger 2013-04-03 10:47:06.719287 7fcdffcdb700 0 log [INF] : pgmap v29136: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 8673B/s wr, 0op/s 2013-04-03 10:47:06.719346 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos 2013-04-03 10:47:06.719365 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 2013-04-03 10:47:06.719383 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:06.719428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos 2013-04-03 10:47:06.719443 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 2013-04-03 10:47:06.719452 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:06.719464 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:47:06.719487 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:06.719499 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:47:06.719509 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 logging 2013-04-03 10:47:06.719288 mon.0 10.252.24.25:6789/0 202 : [INF] pgmap v29136: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 8673B/s wr, 0op/s 2013-04-03 10:47:06.731905 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos 2013-04-03 10:47:06.731922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 2013-04-03 10:47:06.731937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:07.600083 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140490 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:47:07.719009 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394124 encode_pending v394125 2013-04-03 10:47:07.912896 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos 2013-04-03 10:47:07.912922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394124 2013-04-03 10:47:07.912939 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos latest full 393798 2013-04-03 10:47:07.912956 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394125 update_from_paxos applying incremental log 394125 2013-04-03 10:47:06.719288 mon.0 10.252.24.25:6789/0 202 : [INF] pgmap v29136: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 8673B/s wr, 0op/s 2013-04-03 10:47:07.913000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 check_subs 2013-04-03 10:47:07.913032 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 create_pending v 394126 2013-04-03 10:47:07.913054 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394125 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:07.913115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos 2013-04-03 10:47:07.913131 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 2013-04-03 10:47:07.913147 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:07.925580 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos 2013-04-03 10:47:07.925597 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 2013-04-03 10:47:07.925611 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:08.382285 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29136 check_down_pgs 2013-04-03 10:47:08.382322 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29136 v29136: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 8673B/s wr, 0op/s 2013-04-03 10:47:08.382351 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:08.382373 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:47:08.382410 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:47:08.382441 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos 2013-04-03 10:47:08.382459 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 2013-04-03 10:47:08.382469 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394125 log 2013-04-03 10:47:08.382486 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:08.382505 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:47:08.946808 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:08.946818 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a51dc0 from 10.252.24.88:6789/0 2013-04-03 10:47:08.946875 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 205 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:47:09.397747 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21861 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:47:10.913724 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos 2013-04-03 10:47:10.913744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 2013-04-03 10:47:10.913761 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:11.321220 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x300c000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33180/0) 2013-04-03 10:47:11.600225 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140491 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:47:12.946965 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:12.946975 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x21452c0 from 10.252.24.88:6789/0 2013-04-03 10:47:12.947029 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 206 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:47:13.289141 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:13.289153 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 2013-04-03 10:47:13.289204 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 preprocess_query pg_stats(1 pgs tid 47 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:13.289228 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 prepare_update pg_stats(1 pgs tid 47 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:13.289234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 prepare_pg_stats pg_stats(1 pgs tid 47 v 605) v1 from osd.2 2013-04-03 10:47:13.289240 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:47:13.339421 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29136 encode_pending v 29137 2013-04-03 10:47:13.368795 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29136 update_from_paxos applying incremental 29137 2013-04-03 10:47:13.368810 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 v29137: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1985B/s wr, 0op/s 2013-04-03 10:47:13.368828 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 send_pg_creates to 0 pgs 2013-04-03 10:47:13.368830 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 update_logger 2013-04-03 10:47:13.369115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 create_pending v 29138 2013-04-03 10:47:13.369131 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29137 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:13.369149 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 check_osd_map already seen 605 >= 605 2013-04-03 10:47:13.369152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 update_logger 2013-04-03 10:47:13.369163 7fcdffcdb700 0 log [INF] : pgmap v29137: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1985B/s wr, 0op/s 2013-04-03 10:47:13.369223 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos 2013-04-03 10:47:13.369243 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 2013-04-03 10:47:13.369259 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:13.369300 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos 2013-04-03 10:47:13.369315 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 2013-04-03 10:47:13.369323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:13.369333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:47:13.369355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:13.369367 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:47:13.369376 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 logging 2013-04-03 10:47:13.369164 mon.0 10.252.24.25:6789/0 203 : [INF] pgmap v29137: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1985B/s wr, 0op/s 2013-04-03 10:47:13.381424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos 2013-04-03 10:47:13.381441 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 2013-04-03 10:47:13.381456 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:13.382617 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29137 check_down_pgs 2013-04-03 10:47:13.382646 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29137 v29137: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1985B/s wr, 0op/s 2013-04-03 10:47:13.382674 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:13.382695 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:47:13.382732 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:47:13.382765 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos 2013-04-03 10:47:13.382789 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 2013-04-03 10:47:13.382797 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394125 log 2013-04-03 10:47:13.382812 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:13.382828 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:47:13.397802 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21862 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:47:14.368596 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394125 encode_pending v394126 2013-04-03 10:47:14.397265 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos 2013-04-03 10:47:14.397292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394125 2013-04-03 10:47:14.397307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos latest full 393798 2013-04-03 10:47:14.397323 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394126 update_from_paxos applying incremental log 394126 2013-04-03 10:47:13.369164 mon.0 10.252.24.25:6789/0 203 : [INF] pgmap v29137: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1985B/s wr, 0op/s 2013-04-03 10:47:14.397371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 check_subs 2013-04-03 10:47:14.397403 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 create_pending v 394127 2013-04-03 10:47:14.397425 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394126 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:14.397489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos 2013-04-03 10:47:14.397505 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394126 2013-04-03 10:47:14.397526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:14.408633 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos 2013-04-03 10:47:14.408650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394126 2013-04-03 10:47:14.408665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:15.600314 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140492 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:47:16.947050 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:16.947060 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2145b00 from 10.252.24.88:6789/0 2013-04-03 10:47:16.947114 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 207 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:47:17.397942 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21863 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:47:17.398056 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos 2013-04-03 10:47:17.398076 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394126 2013-04-03 10:47:17.398093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:18.289901 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:18.289912 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 2013-04-03 10:47:18.289964 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 preprocess_query pg_stats(1 pgs tid 48 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:18.289988 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 prepare_update pg_stats(1 pgs tid 48 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:18.289994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 prepare_pg_stats pg_stats(1 pgs tid 48 v 605) v1 from osd.2 2013-04-03 10:47:18.290000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:47:18.340186 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29137 encode_pending v 29138 2013-04-03 10:47:18.384252 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29137 update_from_paxos applying incremental 29138 2013-04-03 10:47:18.384267 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 v29138: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2461B/s wr, 0op/s 2013-04-03 10:47:18.384285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 send_pg_creates to 0 pgs 2013-04-03 10:47:18.384287 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 update_logger 2013-04-03 10:47:18.384297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 create_pending v 29139 2013-04-03 10:47:18.384309 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29138 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:18.384324 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 check_osd_map already seen 605 >= 605 2013-04-03 10:47:18.384326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 update_logger 2013-04-03 10:47:18.384336 7fcdffcdb700 0 log [INF] : pgmap v29138: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2461B/s wr, 0op/s 2013-04-03 10:47:18.384393 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos 2013-04-03 10:47:18.384413 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394126 2013-04-03 10:47:18.384431 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:18.384473 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29138 check_down_pgs 2013-04-03 10:47:18.384491 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29138 v29138: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2461B/s wr, 0op/s 2013-04-03 10:47:18.384515 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:18.384537 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:47:18.384573 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:47:18.384606 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos 2013-04-03 10:47:18.384622 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394126 2013-04-03 10:47:18.384632 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394126 log 2013-04-03 10:47:18.384647 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:18.384664 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth 2013-04-03 10:47:18.384698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos 2013-04-03 10:47:18.384713 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394126 2013-04-03 10:47:18.384722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:18.384733 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:47:18.384756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:18.384768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:47:18.384785 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 logging 2013-04-03 10:47:18.384337 mon.0 10.252.24.25:6789/0 204 : [INF] pgmap v29138: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2461B/s wr, 0op/s 2013-04-03 10:47:18.399099 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos 2013-04-03 10:47:18.399116 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394126 2013-04-03 10:47:18.399131 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:19.377854 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394126 encode_pending v394127 2013-04-03 10:47:19.421916 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos 2013-04-03 10:47:19.421945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394126 2013-04-03 10:47:19.421959 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos latest full 393798 2013-04-03 10:47:19.421976 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394127 update_from_paxos applying incremental log 394127 2013-04-03 10:47:18.384337 mon.0 10.252.24.25:6789/0 204 : [INF] pgmap v29138: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2461B/s wr, 0op/s 2013-04-03 10:47:19.422022 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 check_subs 2013-04-03 10:47:19.422052 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 create_pending v 394128 2013-04-03 10:47:19.422074 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394127 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:19.422139 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos 2013-04-03 10:47:19.422155 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 2013-04-03 10:47:19.422174 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:19.434630 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos 2013-04-03 10:47:19.434646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 2013-04-03 10:47:19.434661 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:19.600460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140493 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:47:20.745452 7fce026be700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56643 s=2 pgs=151511 cs=1 l=0).fault with nothing to send, going to standby 2013-04-03 10:47:20.745743 7fcdfe5d2700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.25:0/3745741276 client protocol 0 2013-04-03 10:47:20.745871 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:47:20.745876 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11041 10.252.24.25:0/3745741276 is open for client.11041 10.252.24.25:0/3745741276 2013-04-03 10:47:20.745881 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:47:20.745924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:20.745961 7fcdfe6d3700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.30:0/802047285 client protocol 0 2013-04-03 10:47:20.746396 7fcdfe2cf700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.32:0/3871608469 client protocol 0 2013-04-03 10:47:20.745939 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11041 10.252.24.25:0/3745741276 2013-04-03 10:47:20.746514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=30 2013-04-03 10:47:20.746676 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:47:20.746680 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11074 10.252.24.32:0/3871608469 is open for client.11074 10.252.24.32:0/3871608469 2013-04-03 10:47:20.746684 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:47:20.746706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:20.746719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11074 10.252.24.32:0/3871608469 2013-04-03 10:47:20.746731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=30 2013-04-03 10:47:20.746776 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:47:20.746778 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11051 10.252.24.30:0/802047285 is open for client.11051 10.252.24.30:0/802047285 2013-04-03 10:47:20.746782 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:47:20.746800 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:20.746814 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11051 10.252.24.30:0/802047285 2013-04-03 10:47:20.746823 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=30 2013-04-03 10:47:20.746879 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:20.746895 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.11041 10.252.24.25:0/3745741276 2013-04-03 10:47:20.746906 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=128 2013-04-03 10:47:20.747270 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+,osdmap=606}) v2 2013-04-03 10:47:20.747279 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:20.747407 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:20.747424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.11051 10.252.24.30:0/802047285 2013-04-03 10:47:20.747435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=128 2013-04-03 10:47:20.747603 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:20.747619 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.11074 10.252.24.32:0/3871608469 2013-04-03 10:47:20.747631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=128 2013-04-03 10:47:20.748101 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 2013-04-03 10:47:20.748116 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:20.748445 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 2013-04-03 10:47:20.748457 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:20.748988 7fcdfe7d4700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.88:6801/17538 osd protocol 0 2013-04-03 10:47:20.749305 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:47:20.749314 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: osd.3 10.252.24.88:6801/17538 is open for osd.3 10.252.24.88:6801/17538 2013-04-03 10:47:20.749323 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:47:20.749345 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:20.749359 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 0 26 bytes epoch 5) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:47:20.749371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=26 2013-04-03 10:47:20.749921 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:20.749937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:47:20.749949 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=128 2013-04-03 10:47:20.750531 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=603}) v2 2013-04-03 10:47:20.750537 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:20.750546 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e605 send_incremental [603..605] to osd.3 10.252.24.88:6801/17538 2013-04-03 10:47:20.750570 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 build_incremental [603..605] 2013-04-03 10:47:20.750722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 preprocess_query pg_stats(1 pgs tid 30464 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:47:20.750747 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 prepare_update pg_stats(1 pgs tid 30464 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:47:20.750753 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 prepare_pg_stats pg_stats(1 pgs tid 30464 v 605) v1 from osd.3 2013-04-03 10:47:20.750759 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:47:20.750795 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=603}) v2 2013-04-03 10:47:20.750800 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:20.750804 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e605 send_incremental [603..605] to osd.3 10.252.24.88:6801/17538 2013-04-03 10:47:20.750818 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 build_incremental [603..605] 2013-04-03 10:47:20.801508 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29138 encode_pending v 29139 2013-04-03 10:47:20.804210 7fce027bf700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56643 s=1 pgs=151511 cs=2 l=0).fault 2013-04-03 10:47:21.398086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21864 v187) v2 from mds.? 10.252.24.88:6800/17213 2013-04-03 10:47:22.225006 7fce027bf700 10 mon.hotcpc9039@0(leader) e5 ms_get_authorizer for mon 2013-04-03 10:47:22.225798 7fce026be700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.88:6789/0 mon protocol 2 2013-04-03 10:47:22.226206 7fce026be700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.88:6789/0 mon protocol 2 2013-04-03 10:47:22.226304 7fce026be700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=0 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 2 state connecting 2013-04-03 10:47:22.226314 7fce026be700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=0 pgs=0 cs=0 l=0).accept peer reset, then tried to connect to us, replacing 2013-04-03 10:47:22.226760 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc12533) v4 2013-04-03 10:47:22.226770 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_probe_probe mon.2 10.252.24.88:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc12533) v4 2013-04-03 10:47:22.266007 7fcdffcdb700 5 mon.hotcpc9039@0(leader).elector(352) handle_propose from mon.2 2013-04-03 10:47:22.266012 7fcdffcdb700 10 mon.hotcpc9039@0(leader).elector(352) bump_epoch 352 to 353 2013-04-03 10:47:22.267655 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 reset 2013-04-03 10:47:22.267671 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish 2013-04-03 10:47:22.267726 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 start_election 2013-04-03 10:47:22.267730 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 cancel_probe_timeout (none scheduled) 2013-04-03 10:47:22.267736 7fcdffcdb700 0 log [INF] : mon.hotcpc9039 calling new monitor election 2013-04-03 10:47:22.267756 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(353) start -- can i be leader? 2013-04-03 10:47:22.267770 7fcdffcdb700 1 mon.hotcpc9039@0(electing).elector(353) init, last seen epoch 353 2013-04-03 10:47:22.268621 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(353) handle_ack from mon.2 2013-04-03 10:47:22.268624 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(353) so far i have {0=4294967295,2=4294967295} 2013-04-03 10:47:22.296662 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 ms_handle_reset 0x200d420 10.252.24.88:6800/17213 2013-04-03 10:47:22.296671 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 reset/close on session mds.? 10.252.24.88:6800/17213 2013-04-03 10:47:22.296678 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 remove_session MonSession: mds.? 10.252.24.88:6800/17213 is openallow rwx mds.? 10.252.24.88:6800/17213 2013-04-03 10:47:22.964348 7fcdfead7700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.24.30:6801/5656 osd protocol 0 2013-04-03 10:47:22.964742 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 26 bytes epoch 5) v1 until we get in quorum 2013-04-03 10:47:23.384816 7fce004dc700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 26 bytes epoch 5) v1 until we get in quorum 2013-04-03 10:47:23.431614 7fcdfdfcc700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.24.88:6800/18040 mds protocol 0 2013-04-03 10:47:23.431968 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 36 bytes epoch 0) v1 until we get in quorum 2013-04-03 10:47:24.061895 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 ms_handle_reset 0x319b160 10.252.24.88:6801/17538 2013-04-03 10:47:24.061909 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 reset/close on session osd.3 10.252.24.88:6801/17538 2013-04-03 10:47:24.061913 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 remove_session MonSession: osd.3 10.252.24.88:6801/17538 is openallow rwx osd.3 10.252.24.88:6801/17538 2013-04-03 10:47:24.062075 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+,osdmap=606}) v2 2013-04-03 10:47:24.062082 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:24.449410 7fcdfe7d4700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.24.30:0/517 client protocol 0 2013-04-03 10:47:24.449749 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 30 bytes epoch 5) v1 until we get in quorum 2013-04-03 10:47:24.538078 7fcdfddca700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.24.25:0/2614196857 client protocol 0 2013-04-03 10:47:24.538085 7fcdfdcc9700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.24.25:0/2151250553 client protocol 0 2013-04-03 10:47:24.538221 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 30 bytes epoch 5) v1 until we get in quorum 2013-04-03 10:47:24.538232 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 30 bytes epoch 5) v1 until we get in quorum 2013-04-03 10:47:24.695111 7fcdfd9c6700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.58.31:6802/2682 osd protocol 0 2013-04-03 10:47:24.695464 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 26 bytes epoch 5) v1 until we get in quorum 2013-04-03 10:47:24.860538 7fcdfd7c4700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.24.32:0/1396772501 client protocol 0 2013-04-03 10:47:24.860851 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 30 bytes epoch 5) v1 until we get in quorum 2013-04-03 10:47:25.964036 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 ms_handle_reset 0x3311c60 10.252.24.30:6801/5656 2013-04-03 10:47:26.431177 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 ms_handle_reset 0x3311b00 10.252.24.88:6800/18040 2013-04-03 10:47:27.267869 7fce004dc700 5 mon.hotcpc9039@0(electing).elector(353) election timer expired 2013-04-03 10:47:27.267890 7fce004dc700 10 mon.hotcpc9039@0(electing).elector(353) bump_epoch 353 to 354 2013-04-03 10:47:27.270622 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 win_election, epoch 354 quorum is 0,2 features are 4294967295 2013-04-03 10:47:27.270632 7fce004dc700 0 log [INF] : mon.hotcpc9039@0 won leader election with quorum 0,2 2013-04-03 10:47:27.271064 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish 2013-04-03 10:47:27.271069 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:47:27.271072 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: osd.2 10.252.24.30:6801/5656 is open for osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:27.271077 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:47:27.271108 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:47:27.271110 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: mds.? 10.252.24.88:6800/18040 is open for mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:27.271115 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:47:27.271140 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:47:27.271144 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11802 10.252.24.30:0/517 is open for client.11802 10.252.24.30:0/517 2013-04-03 10:47:27.271148 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:47:27.271174 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:47:27.271177 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11042 10.252.24.25:0/2614196857 is open for client.11042 10.252.24.25:0/2614196857 2013-04-03 10:47:27.271181 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:47:27.271208 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:47:27.271211 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11045 10.252.24.25:0/2151250553 is open for client.11045 10.252.24.25:0/2151250553 2013-04-03 10:47:27.271216 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:47:27.271244 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:47:27.271247 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: osd.0 10.252.58.31:6802/2682 is open for osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:27.271251 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:47:27.271277 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:47:27.271279 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11071 10.252.24.32:0/1396772501 is open for client.11071 10.252.24.32:0/1396772501 2013-04-03 10:47:27.271283 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:47:27.271311 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 resend_routed_requests 2013-04-03 10:47:27.271313 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 register_cluster_logger - already registered 2013-04-03 10:47:27.271316 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start 2013-04-03 10:47:27.271317 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round curr 0 2013-04-03 10:47:27.271318 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round new 1 2013-04-03 10:47:27.271320 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck 2013-04-03 10:47:27.271321 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck start timecheck epoch 354 round 1 2013-04-03 10:47:27.271325 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck send time_check( ping e 354 r 1 ) v1 to mon.2 10.252.24.88:6789/0 2013-04-03 10:47:27.271336 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round setting up next event 2013-04-03 10:47:27.449126 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_handle_reset 0x33119a0 10.252.24.30:0/517 2013-04-03 10:47:27.449139 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 reset/close on session client.11802 10.252.24.30:0/517 2013-04-03 10:47:27.449145 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 remove_session MonSession: client.11802 10.252.24.30:0/517 is open client.11802 10.252.24.30:0/517 2013-04-03 10:47:27.464408 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck time_check( pong e 354 r 1 ts 2013-04-03 10:47:27.462969 ) v1 2013-04-03 10:47:27.464413 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader time_check( pong e 354 r 1 ts 2013-04-03 10:47:27.462969 ) v1 2013-04-03 10:47:27.464420 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader from mon.2 10.252.24.88:6789/0 ts 2013-04-03 10:47:27.462969 delta -0.00144696 skew_bound 0 latency 0.193092 2013-04-03 10:47:27.464432 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader got pongs from everybody (2 total) 2013-04-03 10:47:27.464434 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish_round curr 1 2013-04-03 10:47:27.464436 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_report 2013-04-03 10:47:27.464440 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_report send report to mon.2 10.252.24.88:6789/0 2013-04-03 10:47:27.482783 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29138 update_from_paxos applying incremental 29139 2013-04-03 10:47:27.482802 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 v29139: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 686B/s wr, 0op/s 2013-04-03 10:47:27.482821 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 send_pg_creates to 0 pgs 2013-04-03 10:47:27.482825 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 update_logger 2013-04-03 10:47:27.482835 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 create_pending v 29140 2013-04-03 10:47:27.483202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 check_osd_map already seen 605 >= 605 2013-04-03 10:47:27.483205 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 update_logger 2013-04-03 10:47:27.483217 7fcdffcdb700 0 log [INF] : pgmap v29139: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 686B/s wr, 0op/s 2013-04-03 10:47:27.483266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 create_pending e188 2013-04-03 10:47:27.483285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:27.483296 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 update_logger 2013-04-03 10:47:27.483301 7fcdffcdb700 0 log [INF] : mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:27.483340 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 create_pending e 606 2013-04-03 10:47:27.483349 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 remove_redundant_pg_temp 2013-04-03 10:47:27.483351 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 remove_down_pg_temp 2013-04-03 10:47:27.483371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 update_logger 2013-04-03 10:47:27.483377 7fcdffcdb700 0 log [INF] : osdmap e605: 5 osds: 5 up, 5 in 2013-04-03 10:47:27.483393 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos 2013-04-03 10:47:27.483412 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 2013-04-03 10:47:27.483437 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 create_pending v 394128 2013-04-03 10:47:27.483475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).monmap v5 create_pending monmap epoch 6 2013-04-03 10:47:27.483498 7fcdffcdb700 0 log [INF] : monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} 2013-04-03 10:47:27.483522 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:27.483548 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 create_pending v 2062 2013-04-03 10:47:27.483569 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 AuthMonitor::on_active() 2013-04-03 10:47:27.483594 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos 2013-04-03 10:47:27.483608 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 2013-04-03 10:47:27.483617 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:27.483628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:47:27.483646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:27.483657 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:47:27.483667 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:22.267737 mon.0 10.252.24.25:6789/0 205 : [INF] mon.hotcpc9039 calling new monitor election 2013-04-03 10:47:27.483713 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140494 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:47:27.483754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 preprocess_query pg_stats(1 pgs tid 30464 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:47:27.483761 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 PGMonitor::preprocess_pg_stats: no monitor session! 2013-04-03 10:47:27.483783 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 preprocess_query pg_stats(1 pgs tid 30465 v 605) v1 from osd.3 10.252.24.88:6801/17538 2013-04-03 10:47:27.483789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 PGMonitor::preprocess_pg_stats: no monitor session! 2013-04-03 10:47:27.483806 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:27.483821 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 0 26 bytes epoch 5) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:27.483834 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=26 2013-04-03 10:47:27.483878 7fcdffcdb700 0 -- 10.252.24.25:6789/0 submit_message auth_reply(proto 2 0 Success) v1 remote, 10.252.24.30:6801/5656, failed lossy con, dropping message 0x1c16a00 2013-04-03 10:47:27.483903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos 2013-04-03 10:47:27.483919 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 0 36 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:27.483930 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=36 2013-04-03 10:47:27.483942 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 AuthMonitor::assign_global_id m=auth(proto 0 36 bytes epoch 0) v1 mon=0/3 last_allocated=11896 max_global_id=11896 2013-04-03 10:47:27.483953 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 next_global_id should be 11898 2013-04-03 10:47:27.483962 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prepare_update auth(proto 0 36 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:27.483974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=36 2013-04-03 10:47:27.483984 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 AuthMonitor::assign_global_id m=auth(proto 0 36 bytes epoch 0) v1 mon=0/3 last_allocated=11896 max_global_id=11896 2013-04-03 10:47:27.483993 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 next_global_id should be 11898 2013-04-03 10:47:27.484001 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 increasing max_global_id to 11996 2013-04-03 10:47:27.484009 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 increasing global id, waitlisting message 2013-04-03 10:47:27.484056 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 encode_pending v 2062 2013-04-03 10:47:27.485704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos 2013-04-03 10:47:27.485720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 2013-04-03 10:47:27.485732 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:27.485743 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:47:27.485764 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 nothing new 2013-04-03 10:47:27.485797 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos 2013-04-03 10:47:27.485812 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 2013-04-03 10:47:27.485821 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:27.485831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:47:27.485858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:27.485870 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:47:27.485880 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:22.267737 mon.0 10.252.24.25:6789/0 205 : [INF] mon.hotcpc9039 calling new monitor election 2013-04-03 10:47:27.485900 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:27.270635 mon.0 10.252.24.25:6789/0 206 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 2013-04-03 10:47:27.485950 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos 2013-04-03 10:47:27.485964 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 2013-04-03 10:47:27.485973 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:27.485983 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:47:27.486006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:27.486017 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:47:27.486033 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:27.270635 mon.0 10.252.24.25:6789/0 206 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 2013-04-03 10:47:27.486053 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:27.483218 mon.0 10.252.24.25:6789/0 207 : [INF] pgmap v29139: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 686B/s wr, 0op/s 2013-04-03 10:47:27.486101 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos 2013-04-03 10:47:27.486116 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 2013-04-03 10:47:27.486125 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:27.486135 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:47:27.486158 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:27.486168 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:47:27.486178 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:27.483218 mon.0 10.252.24.25:6789/0 207 : [INF] pgmap v29139: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 686B/s wr, 0op/s 2013-04-03 10:47:27.486197 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:27.483302 mon.0 10.252.24.25:6789/0 208 : [INF] mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:27.486246 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos 2013-04-03 10:47:27.486260 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 2013-04-03 10:47:27.486269 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:27.486279 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:47:27.486302 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:27.486314 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:47:27.486323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:27.483302 mon.0 10.252.24.25:6789/0 208 : [INF] mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:27.486348 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:27.483377 mon.0 10.252.24.25:6789/0 209 : [INF] osdmap e605: 5 osds: 5 up, 5 in 2013-04-03 10:47:27.526736 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mon.2 10.252.24.88:6789/0 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:27.526744 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24ebe00 from 10.252.24.88:6789/0 2013-04-03 10:47:27.526775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos 2013-04-03 10:47:27.526791 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 2013-04-03 10:47:27.526801 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_query log(1 entries) v1 from mon.2 10.252.24.88:6789/0 2013-04-03 10:47:27.526811 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_log log(1 entries) v1 from mon.2 2013-04-03 10:47:27.526824 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_update log(1 entries) v1 from mon.2 10.252.24.88:6789/0 2013-04-03 10:47:27.526836 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_log log(1 entries) v1 from mon.2 2013-04-03 10:47:27.526852 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:22.226658 mon.2 10.252.24.88:6789/0 1 : [INF] mon.hotcpc12533 calling new monitor election 2013-04-03 10:47:27.538165 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_handle_reset 0x3311840 10.252.24.25:0/2614196857 2013-04-03 10:47:27.538173 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 reset/close on session client.11042 10.252.24.25:0/2614196857 2013-04-03 10:47:27.538177 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 remove_session MonSession: client.11042 10.252.24.25:0/2614196857 is open client.11042 10.252.24.25:0/2614196857 2013-04-03 10:47:27.538224 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_handle_reset 0x33116e0 10.252.24.25:0/2151250553 2013-04-03 10:47:27.538230 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 reset/close on session client.11045 10.252.24.25:0/2151250553 2013-04-03 10:47:27.538233 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 remove_session MonSession: client.11045 10.252.24.25:0/2151250553 is open client.11045 10.252.24.25:0/2151250553 2013-04-03 10:47:27.539817 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos 2013-04-03 10:47:27.539853 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos version 2062 keys ver 2061 latest 2059 2013-04-03 10:47:27.539863 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos key server version 2061 2013-04-03 10:47:27.539880 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos() last_allocated_id=11896 max_global_id=11996 2013-04-03 10:47:27.539896 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 create_pending v 2063 2013-04-03 10:47:27.539928 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos 2013-04-03 10:47:27.539943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11802 10.252.24.30:0/517 2013-04-03 10:47:27.539956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 prep_auth() blob_size=30 2013-04-03 10:47:27.539965 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 no session, dropping 2013-04-03 10:47:27.539986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos 2013-04-03 10:47:27.540001 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11042 10.252.24.25:0/2614196857 2013-04-03 10:47:27.540012 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 prep_auth() blob_size=30 2013-04-03 10:47:27.540021 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 no session, dropping 2013-04-03 10:47:27.540041 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos 2013-04-03 10:47:27.540058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11045 10.252.24.25:0/2151250553 2013-04-03 10:47:27.540070 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 prep_auth() blob_size=30 2013-04-03 10:47:27.540078 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 no session, dropping 2013-04-03 10:47:27.540098 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos 2013-04-03 10:47:27.540113 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 preprocess_query auth(proto 0 26 bytes epoch 5) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:27.540124 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 prep_auth() blob_size=26 2013-04-03 10:47:27.540178 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos 2013-04-03 10:47:27.540194 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11071 10.252.24.32:0/1396772501 2013-04-03 10:47:27.540206 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 prep_auth() blob_size=30 2013-04-03 10:47:27.540257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos 2013-04-03 10:47:27.540273 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 preprocess_query global_id (11896) v1 from mon.2 10.252.24.88:6789/0 2013-04-03 10:47:27.540285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 prepare_update global_id (11896) v1 from mon.2 10.252.24.88:6789/0 2013-04-03 10:47:27.540297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 AuthMonitor::prepare_global_id 2013-04-03 10:47:27.540305 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 increasing max_global_id to 12096 2013-04-03 10:47:27.540706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 encode_pending v 2063 2013-04-03 10:47:27.590148 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:47:27.590178 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos version 2063 keys ver 2062 latest 2059 2013-04-03 10:47:27.590187 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos key server version 2062 2013-04-03 10:47:27.590202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos() last_allocated_id=11896 max_global_id=12096 2013-04-03 10:47:27.590218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 create_pending v 2064 2013-04-03 10:47:27.590248 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:47:27.590261 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 36 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:27.590272 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=36 2013-04-03 10:47:27.590280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 AuthMonitor::assign_global_id m=auth(proto 0 36 bytes epoch 0) v1 mon=0/3 last_allocated=11896 max_global_id=12096 2013-04-03 10:47:27.590289 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 next_global_id should be 11898 2013-04-03 10:47:27.590304 7fcdffcdb700 0 -- 10.252.24.25:6789/0 submit_message auth_reply(proto 0 -22 Invalid argument) v1 remote, 10.252.24.88:6800/18040, failed lossy con, dropping message 0x2b78400 2013-04-03 10:47:27.590325 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:47:27.590339 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.11071 10.252.24.32:0/1396772501 2013-04-03 10:47:27.590348 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 2013-04-03 10:47:27.590508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:47:27.590522 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:27.590533 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 2013-04-03 10:47:27.590658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 AuthMonitor::on_active() 2013-04-03 10:47:27.590715 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos 2013-04-03 10:47:27.590730 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 2013-04-03 10:47:27.590747 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:47:27.591193 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 2013-04-03 10:47:27.591203 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:27.591222 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=604}) v2 2013-04-03 10:47:27.591225 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:27.591231 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e605 send_incremental [604..605] to osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:27.591250 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 build_incremental [604..605] 2013-04-03 10:47:27.591312 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=604}) v2 2013-04-03 10:47:27.591317 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:27.591320 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e605 send_incremental [604..605] to osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:27.591335 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 build_incremental [604..605] 2013-04-03 10:47:27.600632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140495 v187) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:47:27.601841 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos 2013-04-03 10:47:27.601859 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 2013-04-03 10:47:27.601874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:47:27.846229 7fcdfdcc9700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.88:6804/18376 osd protocol 0 2013-04-03 10:47:27.846590 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:47:27.846596 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: osd.3 10.252.24.88:6804/18376 is open for osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:27.846601 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:47:27.846644 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:47:27.846661 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 26 bytes epoch 0) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:27.846673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=26 2013-04-03 10:47:27.846687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 AuthMonitor::assign_global_id m=auth(proto 0 26 bytes epoch 0) v1 mon=0/3 last_allocated=11898 max_global_id=12096 2013-04-03 10:47:27.846699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 next_global_id should be 11901 2013-04-03 10:47:27.847515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:47:27.847532 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 32 bytes epoch 0) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:27.847544 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=32 2013-04-03 10:47:27.848301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:47:27.848317 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 165 bytes epoch 0) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:27.848328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=165 2013-04-03 10:47:27.849105 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=0+}) v2 2013-04-03 10:47:27.849113 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 0 have 5 2013-04-03 10:47:27.849146 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 2013-04-03 10:47:27.849149 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 0 have 5 2013-04-03 10:47:27.849193 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:47:27.849209 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 2 bytes epoch 0) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:27.849220 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=2 2013-04-03 10:47:27.850372 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_get_version mon_get_version(what=osdmap handle=1) v1 2013-04-03 10:47:27.850999 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 preprocess_query osd_boot(osd.3 booted 0 v605) v3 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:27.851007 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 preprocess_boot from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:27.851011 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e605 prepare_update osd_boot(osd.3 booted 0 v605) v3 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:27.851014 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e605 prepare_boot from osd.3 10.252.24.88:6804/18376 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.3 0b7e2707-d3a7-4f74-93fa-0686f3188d92 e605 [104,605] lci=[576,605]) cluster_addr 10.252.24.88:6805/18376 hb_addr 10.252.24.88:6806/18376 2013-04-03 10:47:27.851023 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e605 prepare_boot was up, first marking down osd.3 10.252.24.88:6801/17538 2013-04-03 10:47:27.851027 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 should_propose 2013-04-03 10:47:28.384969 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29139 check_down_pgs 2013-04-03 10:47:28.384993 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29139 v29139: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 686B/s wr, 0op/s 2013-04-03 10:47:28.385022 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:28.385047 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in 2013-04-03 10:47:28.385083 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 2013-04-03 10:47:28.385113 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos 2013-04-03 10:47:28.385132 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 2013-04-03 10:47:28.385141 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394127 log 2013-04-03 10:47:28.385158 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:47:28.385180 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:47:28.477267 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394127 encode_pending v394128 2013-04-03 10:47:28.559030 7fcdfddca700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.25:6805/3572 osd protocol 0 2013-04-03 10:47:28.559128 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:47:28.559133 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: osd.1 10.252.24.25:6805/3572 is open for osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:28.559138 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:47:28.559162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:47:28.559178 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 26 bytes epoch 5) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:28.559190 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=26 2013-04-03 10:47:28.559424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:47:28.559441 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:28.559453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 2013-04-03 10:47:28.559759 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=603}) v2 2013-04-03 10:47:28.559766 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:28.559771 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e605 send_incremental [603..605] to osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:28.559790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 build_incremental [603..605] 2013-04-03 10:47:28.559880 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 preprocess_query pg_stats(1 pgs tid 69 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:28.559901 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 prepare_update pg_stats(1 pgs tid 69 v 605) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:28.559906 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 prepare_pg_stats pg_stats(1 pgs tid 69 v 605) v1 from osd.1 2013-04-03 10:47:28.559912 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:47:28.559942 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=603}) v2 2013-04-03 10:47:28.559946 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:28.559949 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e605 send_incremental [603..605] to osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:28.559963 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 build_incremental [603..605] 2013-04-03 10:47:28.589946 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29139 encode_pending v 29140 2013-04-03 10:47:28.590039 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 encode_pending e 606 2013-04-03 10:47:28.590043 7fce004dc700 2 mon.hotcpc9039@0(leader).osd e605 osd.3 DOWN 2013-04-03 10:47:28.601552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos 2013-04-03 10:47:28.601705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos version 394128 summary v 394127 2013-04-03 10:47:28.601720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos latest full 393798 2013-04-03 10:47:28.601738 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 update_from_paxos applying incremental log 394128 2013-04-03 10:47:22.226658 mon.2 10.252.24.88:6789/0 1 : [INF] mon.hotcpc12533 calling new monitor election 2013-04-03 10:47:28.601765 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 update_from_paxos applying incremental log 394128 2013-04-03 10:47:22.267737 mon.0 10.252.24.25:6789/0 205 : [INF] mon.hotcpc9039 calling new monitor election 2013-04-03 10:47:28.601785 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 update_from_paxos applying incremental log 394128 2013-04-03 10:47:27.270635 mon.0 10.252.24.25:6789/0 206 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 2013-04-03 10:47:28.602145 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 update_from_paxos applying incremental log 394128 2013-04-03 10:47:27.483218 mon.0 10.252.24.25:6789/0 207 : [INF] pgmap v29139: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 686B/s wr, 0op/s 2013-04-03 10:47:28.602167 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 update_from_paxos applying incremental log 394128 2013-04-03 10:47:27.483302 mon.0 10.252.24.25:6789/0 208 : [INF] mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:28.602186 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 update_from_paxos applying incremental log 394128 2013-04-03 10:47:27.483377 mon.0 10.252.24.25:6789/0 209 : [INF] osdmap e605: 5 osds: 5 up, 5 in 2013-04-03 10:47:28.602224 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 check_subs 2013-04-03 10:47:28.602259 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 create_pending v 394129 2013-04-03 10:47:28.602282 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:28.602301 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:28.602319 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:28.602336 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:28.602353 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:28.602372 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 _updated_log for mon.2 10.252.24.88:6789/0 2013-04-03 10:47:28.638901 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29139 update_from_paxos applying incremental 29140 2013-04-03 10:47:28.638922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 v29140: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 899B/s wr, 0op/s 2013-04-03 10:47:28.638940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 send_pg_creates to 0 pgs 2013-04-03 10:47:28.638943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 update_logger 2013-04-03 10:47:28.638955 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 create_pending v 29141 2013-04-03 10:47:28.638964 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29140 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:28.638981 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 check_osd_map already seen 605 >= 605 2013-04-03 10:47:28.638983 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 update_logger 2013-04-03 10:47:28.638993 7fcdffcdb700 0 log [INF] : pgmap v29140: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 899B/s wr, 0op/s 2013-04-03 10:47:28.639438 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos 2013-04-03 10:47:28.639454 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos version 394128 summary v 394128 2013-04-03 10:47:28.639462 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:28.639471 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:47:28.639501 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:28.639515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:47:28.639525 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 logging 2013-04-03 10:47:27.483501 mon.0 10.252.24.25:6789/0 210 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} 2013-04-03 10:47:28.639544 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 logging 2013-04-03 10:47:28.638994 mon.0 10.252.24.25:6789/0 211 : [INF] pgmap v29140: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 899B/s wr, 0op/s 2013-04-03 10:47:28.677758 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e605 update_from_paxos applying incremental 606 2013-04-03 10:47:28.677803 7fcdffcdb700 1 mon.hotcpc9039@0(leader).osd e606 e606: 5 osds: 4 up, 5 in 2013-04-03 10:47:28.678888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 adding osd.3 to down_pending_out map 2013-04-03 10:47:28.678942 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 check_osd_map applying osdmap e606 to pg_map 2013-04-03 10:47:28.678961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 register_new_pgs checking pg pools for osdmap epoch 606, last_pg_scan 341 2013-04-03 10:47:28.678966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 2013-04-03 10:47:28.678970 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:47:28.678972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:47:28.678974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 2013-04-03 10:47:28.678977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 2013-04-03 10:47:28.678986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 2013-04-03 10:47:28.678989 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 2013-04-03 10:47:28.678994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 2013-04-03 10:47:28.678996 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 2013-04-03 10:47:28.679012 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs 2013-04-03 10:47:28.679014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 check_down_pgs 2013-04-03 10:47:28.679015 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 8.4 stale with acting [3,2] 2013-04-03 10:47:28.679018 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.e stale with acting [3,2] 2013-04-03 10:47:28.679021 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.c stale with acting [3,2] 2013-04-03 10:47:28.679023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 8.2 stale with acting [3,2] 2013-04-03 10:47:28.679026 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.8 stale with acting [3,2] 2013-04-03 10:47:28.679028 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.b stale with acting [3,2] 2013-04-03 10:47:28.679030 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.a stale with acting [3,2] 2013-04-03 10:47:28.679033 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.a stale with acting [3,2] 2013-04-03 10:47:28.679035 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.9 stale with acting [3,2] 2013-04-03 10:47:28.679037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 6.0 stale with acting [3,0] 2013-04-03 10:47:28.679040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 4.2 stale with acting [3,0] 2013-04-03 10:47:28.679042 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.4 stale with acting [3,0] 2013-04-03 10:47:28.679044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.6 stale with acting [3,0] 2013-04-03 10:47:28.679047 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 7.3 stale with acting [3,2] 2013-04-03 10:47:28.679049 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 5.1 stale with acting [3,0] 2013-04-03 10:47:28.679055 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 4.0 stale with acting [3,2] 2013-04-03 10:47:28.679057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 3.7 stale with acting [3,2] 2013-04-03 10:47:28.679060 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.5 stale with acting [3,0] 2013-04-03 10:47:28.679062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.4 stale with acting [3,2] 2013-04-03 10:47:28.679064 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 7.5 stale with acting [3,2] 2013-04-03 10:47:28.679066 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 6.4 stale with acting [3,2] 2013-04-03 10:47:28.679068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 5.7 stale with acting [3,2] 2013-04-03 10:47:28.679071 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 4.6 stale with acting [3,2] 2013-04-03 10:47:28.679073 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 3.1 stale with acting [3,2] 2013-04-03 10:47:28.679075 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.0 stale with acting [3,0] 2013-04-03 10:47:28.679077 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.3 stale with acting [3,2] 2013-04-03 10:47:28.679080 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.2 stale with acting [3,0] 2013-04-03 10:47:28.679082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 6.6 stale with acting [3,2] 2013-04-03 10:47:28.679084 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 5.5 stale with acting [3,2] 2013-04-03 10:47:28.679086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 3.3 stale with acting [3,0] 2013-04-03 10:47:28.679089 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.2 stale with acting [3,2] 2013-04-03 10:47:28.679091 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.1 stale with acting [3,0] 2013-04-03 10:47:28.679094 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.3c stale with acting [3,2] 2013-04-03 10:47:28.679097 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.3b stale with acting [3,2] 2013-04-03 10:47:28.679102 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.3a stale with acting [3,2] 2013-04-03 10:47:28.679105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.31 stale with acting [3,0] 2013-04-03 10:47:28.679107 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.32 stale with acting [3,0] 2013-04-03 10:47:28.679109 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.33 stale with acting [3,0] 2013-04-03 10:47:28.679111 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.30 stale with acting [3,2] 2013-04-03 10:47:28.679114 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.32 stale with acting [3,2] 2013-04-03 10:47:28.679116 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.31 stale with acting [3,2] 2013-04-03 10:47:28.679119 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.24 stale with acting [3,0] 2013-04-03 10:47:28.679122 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.23 stale with acting [3,0] 2013-04-03 10:47:28.679124 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.22 stale with acting [3,0] 2013-04-03 10:47:28.679127 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.1c stale with acting [3,1] 2013-04-03 10:47:28.679129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.1e stale with acting [3,1] 2013-04-03 10:47:28.679131 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.1c stale with acting [3,2] 2013-04-03 10:47:28.679133 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.1d stale with acting [3,2] 2013-04-03 10:47:28.679135 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.1d stale with acting [3,1] 2013-04-03 10:47:28.679138 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.1b stale with acting [3,4] 2013-04-03 10:47:28.679140 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.1a stale with acting [3,4] 2013-04-03 10:47:28.679142 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.19 stale with acting [3,4] 2013-04-03 10:47:28.679144 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.1a stale with acting [3,0] 2013-04-03 10:47:28.679150 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.18 stale with acting [3,0] 2013-04-03 10:47:28.679152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.1b stale with acting [3,2] 2013-04-03 10:47:28.679154 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.19 stale with acting [3,0] 2013-04-03 10:47:28.679157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.16 stale with acting [3,1] 2013-04-03 10:47:28.679159 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.14 stale with acting [3,1] 2013-04-03 10:47:28.679161 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.15 stale with acting [3,4] 2013-04-03 10:47:28.679164 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.14 stale with acting [3,4] 2013-04-03 10:47:28.679166 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.14 stale with acting [3,1] 2013-04-03 10:47:28.679168 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.15 stale with acting [3,1] 2013-04-03 10:47:28.679170 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.13 stale with acting [3,4] 2013-04-03 10:47:28.679173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.12 stale with acting [3,4] 2013-04-03 10:47:28.679175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.11 stale with acting [3,4] 2013-04-03 10:47:28.679177 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.12 stale with acting [3,0] 2013-04-03 10:47:28.679179 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.13 stale with acting [3,1] 2013-04-03 10:47:28.679181 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.10 stale with acting [3,0] 2013-04-03 10:47:28.679184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.13 stale with acting [3,4] 2013-04-03 10:47:28.679186 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.10 stale with acting [3,2] 2013-04-03 10:47:28.679188 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.11 stale with acting [3,0] 2013-04-03 10:47:28.679190 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.12 stale with acting [3,1] 2013-04-03 10:47:28.679196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.f stale with acting [3,2] 2013-04-03 10:47:28.679251 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 encode_pending v 29141 2013-04-03 10:47:28.683727 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 send_pg_creates to 0 pgs 2013-04-03 10:47:28.683731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 send_to_waiting 606 2013-04-03 10:47:28.683734 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e606 send_incremental [606..606] to client.11041 10.252.24.25:0/3745741276 2013-04-03 10:47:28.683754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 build_incremental [606..606] 2013-04-03 10:47:28.683785 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e606 send_incremental [606..606] to client.11074 10.252.24.32:0/3871608469 2013-04-03 10:47:28.683799 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 build_incremental [606..606] 2013-04-03 10:47:28.683826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 committed, telling random osd.2 10.252.24.30:6801/5656 all about it 2013-04-03 10:47:28.683831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 build_incremental [605..606] 2013-04-03 10:47:28.683865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 update_logger 2013-04-03 10:47:28.683882 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 create_pending e 607 2013-04-03 10:47:28.683886 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 remove_redundant_pg_temp 2013-04-03 10:47:28.683889 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 remove_down_pg_temp 2013-04-03 10:47:28.683925 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_query osd_boot(osd.3 booted 0 v605) v3 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:28.683934 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_boot from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:28.683937 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_update osd_boot(osd.3 booted 0 v605) v3 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:28.683941 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_boot from osd.3 10.252.24.88:6804/18376 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.3 0b7e2707-d3a7-4f74-93fa-0686f3188d92 e605 [104,605] lci=[576,605]) cluster_addr 10.252.24.88:6805/18376 hb_addr 10.252.24.88:6806/18376 2013-04-03 10:47:28.683952 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 setting osd.3 uuid to 0b7e2707-d3a7-4f74-93fa-0686f3188d92 2013-04-03 10:47:28.683955 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 old osd_info: up_from 576 up_thru 604 down_at 606 last_clean_interval [541,572) 2013-04-03 10:47:28.683958 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 prepare_boot osd.3 last_clean_interval [541,572) -> [576-605) 2013-04-03 10:47:28.683960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 not laggy, new xi down_stamp 2013-04-03 10:47:28.590043 laggy_probability 0 laggy_interval 0 2013-04-03 10:47:28.683970 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 should_propose 2013-04-03 10:47:28.683982 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 update_logger 2013-04-03 10:47:28.683987 7fcdffcdb700 0 log [INF] : osdmap e606: 5 osds: 4 up, 5 in 2013-04-03 10:47:28.684040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos 2013-04-03 10:47:28.684057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos version 394128 summary v 394128 2013-04-03 10:47:28.684068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:28.684078 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:47:28.684096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 nothing new 2013-04-03 10:47:28.741341 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29140 update_from_paxos applying incremental 29141 2013-04-03 10:47:28.741514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 v29141: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1039B/s wr, 0op/s 2013-04-03 10:47:28.741545 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 send_pg_creates to 0 pgs 2013-04-03 10:47:28.741547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 update_logger 2013-04-03 10:47:28.741563 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 create_pending v 29142 2013-04-03 10:47:28.741571 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 check_osd_map already seen 606 >= 606 2013-04-03 10:47:28.741573 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 update_logger 2013-04-03 10:47:28.741583 7fcdffcdb700 0 log [INF] : pgmap v29141: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1039B/s wr, 0op/s 2013-04-03 10:47:28.741643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos 2013-04-03 10:47:28.741661 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos version 394128 summary v 394128 2013-04-03 10:47:28.741679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:47:28.741721 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos 2013-04-03 10:47:28.741736 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos version 394128 summary v 394128 2013-04-03 10:47:28.741745 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:28.741756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:47:28.741781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:28.741792 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:47:28.741802 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 logging 2013-04-03 10:47:28.638994 mon.0 10.252.24.25:6789/0 211 : [INF] pgmap v29140: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 899B/s wr, 0op/s 2013-04-03 10:47:28.741820 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 logging 2013-04-03 10:47:28.683988 mon.0 10.252.24.25:6789/0 212 : [INF] osdmap e606: 5 osds: 4 up, 5 in 2013-04-03 10:47:28.773870 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos 2013-04-03 10:47:28.773887 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos version 394128 summary v 394128 2013-04-03 10:47:28.773902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:47:28.773928 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:28.773937 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x3306c40 from 10.252.24.88:6789/0 2013-04-03 10:47:28.773960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_query osd_alive(want up_thru 606 have 606) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:47:28.773967 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_alive want up_thru 606 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:47:28.773970 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_update osd_alive(want up_thru 606 have 606) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:47:28.773973 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_alive want up_thru 606 have 606 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:47:28.773977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 should_propose 2013-04-03 10:47:28.897891 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=606}) v2 2013-04-03 10:47:28.897897 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:28.897902 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e606 send_incremental [606..606] to osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:28.898266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 build_incremental [606..606] 2013-04-03 10:47:28.899578 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_query osd_alive(want up_thru 606 have 606) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:28.899585 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_alive want up_thru 606 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:28.899589 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_update osd_alive(want up_thru 606 have 606) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:28.899592 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_alive want up_thru 606 have 606 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:28.899595 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 should_propose 2013-04-03 10:47:28.947253 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:28.947260 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a51000 from 10.252.24.88:6789/0 2013-04-03 10:47:28.947292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 210 v187) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:47:28.967601 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:28.967609 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:47:28.967643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 preprocess_query pg_stats(1 pgs tid 51 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:28.967664 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e606 send_incremental [606..606] to osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:28.967679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 build_incremental [606..606] 2013-04-03 10:47:28.967713 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 prepare_update pg_stats(1 pgs tid 51 v 605) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:28.967718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 prepare_pg_stats pg_stats(1 pgs tid 51 v 605) v1 from osd.2 2013-04-03 10:47:28.967724 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:47:28.968827 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:28.968834 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x336b700 from 10.252.24.88:6789/0 2013-04-03 10:47:28.968865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_query osd_alive(want up_thru 606 have 606) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:28.968874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_alive want up_thru 606 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:28.968878 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_update osd_alive(want up_thru 606 have 606) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:28.968880 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_alive want up_thru 606 have 606 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:28.968884 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 should_propose 2013-04-03 10:47:29.107892 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:47:29.107898 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:47:29.107905 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 2013-04-03 10:47:29.432111 7fcdfe7d4700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 ms_verify_authorizer 10.252.24.88:6800/18040 mds protocol 0 2013-04-03 10:47:29.432467 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 do not have session, making new one 2013-04-03 10:47:29.432474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 ms_dispatch new session MonSession: mds.? 10.252.24.88:6800/18040 is open for mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:29.432481 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 setting timeout on session 2013-04-03 10:47:29.432523 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:29.432540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 preprocess_query auth(proto 0 36 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:29.432553 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 prep_auth() blob_size=36 2013-04-03 10:47:29.432568 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 AuthMonitor::assign_global_id m=auth(proto 0 36 bytes epoch 0) v1 mon=0/3 last_allocated=11901 max_global_id=12096 2013-04-03 10:47:29.432579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 next_global_id should be 11904 2013-04-03 10:47:29.433611 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:29.433628 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 preprocess_query auth(proto 2 32 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:29.433639 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 prep_auth() blob_size=32 2013-04-03 10:47:29.434539 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:29.434555 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 preprocess_query auth(proto 2 181 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:29.434566 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 prep_auth() blob_size=181 2013-04-03 10:47:29.435446 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({monmap=0+}) v2 2013-04-03 10:47:29.435455 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 0 have 5 2013-04-03 10:47:29.435501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:29.435518 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 preprocess_query auth(proto 2 2 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:29.435529 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 prep_auth() blob_size=2 2013-04-03 10:47:29.436331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11904/hotcpc12533 up:boot seq 1 v0) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:29.436341 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 prepare_update mdsbeacon(11904/hotcpc12533 up:boot seq 1 v0) v2 2013-04-03 10:47:29.436345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 fail_mds_gid 11006 mds.hotcpc12533 rank -1 2013-04-03 10:47:29.436348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e606 blacklist 10.252.24.88:6800/17213 until 2013-04-03 11:11:29.436347 2013-04-03 10:47:29.436390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e606 min_last_epoch_clean 593 2013-04-03 10:47:29.436445 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e606 min_last_epoch_clean 593 2013-04-03 10:47:29.436497 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e606 encode_pending e 607 2013-04-03 10:47:29.436501 7fcdffcdb700 2 mon.hotcpc9039@0(leader sync( leader state none )).osd e606 osd.3 UP 10.252.24.88:6804/18376 2013-04-03 10:47:29.438062 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 prepare_beacon pending map now: 2013-04-03 10:47:29.438067 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 print_map epoch 188 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:34:16.921828 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 607 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:active seq 12 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11904: 10.252.24.88:6800/18040 'hotcpc12533' mds.-1.0 up:standby seq 1 2013-04-03 10:47:29.438107 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 encode_pending e188 2013-04-03 10:47:29.438162 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({monmap=6+,osdmap=0}) v2 2013-04-03 10:47:29.438167 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:29.438260 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({mdsmap=0+,monmap=6+,osdmap=0}) v2 2013-04-03 10:47:29.438280 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:29.482277 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e606 update_from_paxos applying incremental 607 2013-04-03 10:47:29.482320 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 e607: 5 osds: 5 up, 5 in 2013-04-03 10:47:29.483430 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 check_osd_map applying osdmap e607 to pg_map 2013-04-03 10:47:29.483456 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 register_new_pgs checking pg pools for osdmap epoch 607, last_pg_scan 341 2013-04-03 10:47:29.483459 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 2013-04-03 10:47:29.483463 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:47:29.483466 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:47:29.483469 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 2013-04-03 10:47:29.483472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 2013-04-03 10:47:29.483475 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 2013-04-03 10:47:29.483477 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 2013-04-03 10:47:29.483480 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 2013-04-03 10:47:29.483483 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 2013-04-03 10:47:29.483491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs 2013-04-03 10:47:29.483494 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 check_down_pgs 2013-04-03 10:47:29.483551 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 encode_pending v 29142 2013-04-03 10:47:29.483852 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 send_pg_creates to 0 pgs 2013-04-03 10:47:29.483862 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_to_waiting 607 2013-04-03 10:47:29.483872 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 committed, telling random osd.0 10.252.58.31:6802/2682 all about it 2013-04-03 10:47:29.483876 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 build_incremental [606..607] 2013-04-03 10:47:29.483921 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 update_logger 2013-04-03 10:47:29.483939 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 create_pending e 608 2013-04-03 10:47:29.483943 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 remove_redundant_pg_temp 2013-04-03 10:47:29.483945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 remove_down_pg_temp 2013-04-03 10:47:29.483963 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 _booted osd.3 10.252.24.88:6804/18376 w 0 from 605 2013-04-03 10:47:29.483980 7fcdffcdb700 0 log [INF] : osd.3 10.252.24.88:6804/18376 boot 2013-04-03 10:47:29.484013 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_latest to osd.3 10.252.24.88:6804/18376 start 606 2013-04-03 10:47:29.484020 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_incremental [606..607] to osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:29.484032 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 build_incremental [606..607] 2013-04-03 10:47:29.484078 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 _reply_map 606 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:47:29.484099 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_latest to osd.4 10.252.24.32:6800/13473 start 606 2013-04-03 10:47:29.484105 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_incremental [606..607] to osd.4 10.252.24.32:6800/13473 2013-04-03 10:47:29.484116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 build_incremental [606..607] 2013-04-03 10:47:29.484158 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 _reply_map 606 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:29.484182 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_latest to osd.0 10.252.58.31:6802/2682 start 606 2013-04-03 10:47:29.484189 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_incremental [606..607] to osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:29.484201 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 build_incremental [606..607] 2013-04-03 10:47:29.484244 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 _reply_map 606 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:29.484263 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_latest to osd.2 10.252.24.30:6801/5656 start 606 2013-04-03 10:47:29.484269 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_incremental [606..607] to osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:29.484280 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 build_incremental [606..607] 2013-04-03 10:47:29.484321 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 update_logger 2013-04-03 10:47:29.484328 7fcdffcdb700 0 log [INF] : osdmap e607: 5 osds: 5 up, 5 in 2013-04-03 10:47:29.484396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 update_from_paxos 2013-04-03 10:47:29.484413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 update_from_paxos version 394128 summary v 394128 2013-04-03 10:47:29.484422 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:29.484434 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:47:29.484465 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:29.484478 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:47:29.484488 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 logging 2013-04-03 10:47:28.683988 mon.0 10.252.24.25:6789/0 212 : [INF] osdmap e606: 5 osds: 4 up, 5 in 2013-04-03 10:47:29.484508 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 logging 2013-04-03 10:47:28.741584 mon.0 10.252.24.25:6789/0 213 : [INF] pgmap v29141: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1039B/s wr, 0op/s 2013-04-03 10:47:29.484560 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 update_from_paxos 2013-04-03 10:47:29.484575 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 update_from_paxos version 394128 summary v 394128 2013-04-03 10:47:29.484584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:29.484597 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:47:29.484620 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:29.484632 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:47:29.484844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 logging 2013-04-03 10:47:28.741584 mon.0 10.252.24.25:6789/0 213 : [INF] pgmap v29141: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1039B/s wr, 0op/s 2013-04-03 10:47:29.484876 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 logging 2013-04-03 10:47:29.483981 mon.0 10.252.24.25:6789/0 214 : [INF] osd.3 10.252.24.88:6804/18376 boot 2013-04-03 10:47:29.485965 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_query osd_alive(want up_thru 607 have 607) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:29.485973 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_alive want up_thru 607 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:29.485977 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_update osd_alive(want up_thru 607 have 607) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:29.485981 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_alive want up_thru 607 have 607 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:29.485985 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 should_propose 2013-04-03 10:47:29.556443 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_query osd_alive(want up_thru 607 have 607) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:29.556451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_alive want up_thru 607 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:29.556455 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_update osd_alive(want up_thru 607 have 607) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:29.556459 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_alive want up_thru 607 have 607 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:29.556462 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 should_propose 2013-04-03 10:47:29.560943 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=606}) v2 2013-04-03 10:47:29.560949 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:29.560954 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_incremental [606..607] to osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:29.560971 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 build_incremental [606..607] 2013-04-03 10:47:29.561699 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_query osd_alive(want up_thru 607 have 607) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:29.561710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_alive want up_thru 607 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:29.561714 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_update osd_alive(want up_thru 607 have 607) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:29.561718 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_alive want up_thru 607 have 607 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:29.561723 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 should_propose 2013-04-03 10:47:29.675387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 update_from_paxos version 188, my e 187 2013-04-03 10:47:29.675405 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 update_from_paxos got 188 2013-04-03 10:47:29.675419 7fcdffcdb700 4 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 new map 2013-04-03 10:47:29.675422 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 print_map epoch 188 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:47:29.438111 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 607 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:active seq 12 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11904: 10.252.24.88:6800/18040 'hotcpc12533' mds.-1.0 up:standby seq 1 2013-04-03 10:47:29.675530 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 update_logger 2013-04-03 10:47:29.675548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 create_pending e189 2013-04-03 10:47:29.675560 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 _updated mds.? mdsbeacon(11904/hotcpc12533 up:boot seq 1 v0) v2 2013-04-03 10:47:29.675570 7fcdffcdb700 0 log [INF] : mds.? 10.252.24.88:6800/18040 up:boot 2013-04-03 10:47:29.675602 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:29.675614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 update_logger 2013-04-03 10:47:29.675620 7fcdffcdb700 0 log [INF] : mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:29.692770 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 update_from_paxos 2013-04-03 10:47:29.692785 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 update_from_paxos version 394128 summary v 394128 2013-04-03 10:47:29.692794 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:29.692804 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:47:29.692847 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:29.692861 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:47:29.692871 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 logging 2013-04-03 10:47:29.483981 mon.0 10.252.24.25:6789/0 214 : [INF] osd.3 10.252.24.88:6804/18376 boot 2013-04-03 10:47:29.692890 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 logging 2013-04-03 10:47:29.484328 mon.0 10.252.24.25:6789/0 215 : [INF] osdmap e607: 5 osds: 5 up, 5 in 2013-04-03 10:47:29.692987 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 encode_pending v394129 2013-04-03 10:47:29.765737 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:29.765746 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x336b540 from 10.252.24.88:6789/0 2013-04-03 10:47:29.765782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_query osd_alive(want up_thru 607 have 607) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:29.765790 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_alive want up_thru 607 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:29.765795 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_update osd_alive(want up_thru 607 have 607) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:29.765800 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_alive want up_thru 607 have 607 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:29.765804 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 should_propose 2013-04-03 10:47:29.778105 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 update_from_paxos applying incremental 29142 2013-04-03 10:47:29.778126 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 v29142: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 9167B/s wr, 1op/s 2013-04-03 10:47:29.778147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 send_pg_creates to 0 pgs 2013-04-03 10:47:29.778151 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 update_logger 2013-04-03 10:47:29.778161 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 create_pending v 29143 2013-04-03 10:47:29.778170 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:29.778187 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 check_osd_map already seen 607 >= 607 2013-04-03 10:47:29.778189 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 update_logger 2013-04-03 10:47:29.778200 7fcdffcdb700 0 log [INF] : pgmap v29142: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 9167B/s wr, 1op/s 2013-04-03 10:47:29.829355 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos 2013-04-03 10:47:29.829383 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394128 2013-04-03 10:47:29.829398 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos latest full 393798 2013-04-03 10:47:29.829416 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos applying incremental log 394129 2013-04-03 10:47:27.483501 mon.0 10.252.24.25:6789/0 210 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} 2013-04-03 10:47:29.829443 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos applying incremental log 394129 2013-04-03 10:47:28.638994 mon.0 10.252.24.25:6789/0 211 : [INF] pgmap v29140: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 899B/s wr, 0op/s 2013-04-03 10:47:29.829463 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos applying incremental log 394129 2013-04-03 10:47:28.683988 mon.0 10.252.24.25:6789/0 212 : [INF] osdmap e606: 5 osds: 4 up, 5 in 2013-04-03 10:47:29.829483 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos applying incremental log 394129 2013-04-03 10:47:28.741584 mon.0 10.252.24.25:6789/0 213 : [INF] pgmap v29141: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1039B/s wr, 0op/s 2013-04-03 10:47:29.829502 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos applying incremental log 394129 2013-04-03 10:47:29.483981 mon.0 10.252.24.25:6789/0 214 : [INF] osd.3 10.252.24.88:6804/18376 boot 2013-04-03 10:47:29.829521 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos applying incremental log 394129 2013-04-03 10:47:29.484328 mon.0 10.252.24.25:6789/0 215 : [INF] osdmap e607: 5 osds: 5 up, 5 in 2013-04-03 10:47:29.829566 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 check_subs 2013-04-03 10:47:29.829601 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 create_pending v 394130 2013-04-03 10:47:29.829626 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:29.829645 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:29.829666 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:29.829683 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:29.829700 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:29.829730 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos 2013-04-03 10:47:29.830094 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 2013-04-03 10:47:29.830108 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:29.830121 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:47:29.830147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:29.830159 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:47:29.830170 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:29.484328 mon.0 10.252.24.25:6789/0 215 : [INF] osdmap e607: 5 osds: 5 up, 5 in 2013-04-03 10:47:29.830189 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:29.675571 mon.0 10.252.24.25:6789/0 216 : [INF] mds.? 10.252.24.88:6800/18040 up:boot 2013-04-03 10:47:29.830232 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos 2013-04-03 10:47:29.830247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 2013-04-03 10:47:29.830256 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:29.830267 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:47:29.830290 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:29.830303 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:47:29.830313 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:29.675571 mon.0 10.252.24.25:6789/0 216 : [INF] mds.? 10.252.24.88:6800/18040 up:boot 2013-04-03 10:47:29.830331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:29.675620 mon.0 10.252.24.25:6789/0 217 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:29.830396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos 2013-04-03 10:47:29.830412 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 2013-04-03 10:47:29.830431 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:29.853461 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos 2013-04-03 10:47:29.853479 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 2013-04-03 10:47:29.853494 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:30.482025 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 min_last_epoch_clean 593 2013-04-03 10:47:30.482093 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 min_last_epoch_clean 593 2013-04-03 10:47:30.482148 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 encode_pending e 608 2013-04-03 10:47:30.512222 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 update_from_paxos applying incremental 608 2013-04-03 10:47:30.512267 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:47:30.513383 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 check_osd_map applying osdmap e608 to pg_map 2013-04-03 10:47:30.513407 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 register_new_pgs checking pg pools for osdmap epoch 608, last_pg_scan 341 2013-04-03 10:47:30.513412 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 2013-04-03 10:47:30.513422 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:47:30.513425 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 2013-04-03 10:47:30.513428 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 2013-04-03 10:47:30.513431 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 2013-04-03 10:47:30.513433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 2013-04-03 10:47:30.513436 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 2013-04-03 10:47:30.513439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 2013-04-03 10:47:30.513444 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 2013-04-03 10:47:30.513453 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs 2013-04-03 10:47:30.513456 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 check_down_pgs 2013-04-03 10:47:30.513514 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 encode_pending v 29143 2013-04-03 10:47:30.513777 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 send_pg_creates to 0 pgs 2013-04-03 10:47:30.513782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_to_waiting 608 2013-04-03 10:47:30.513786 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 committed, telling random osd.1 10.252.24.25:6805/3572 all about it 2013-04-03 10:47:30.513790 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 build_incremental [607..608] 2013-04-03 10:47:30.513837 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 update_logger 2013-04-03 10:47:30.513856 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 create_pending e 609 2013-04-03 10:47:30.513861 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 remove_redundant_pg_temp 2013-04-03 10:47:30.513863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 remove_down_pg_temp 2013-04-03 10:47:30.513882 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 _reply_map 607 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:30.513899 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_latest to osd.0 10.252.58.31:6802/2682 start 607 2013-04-03 10:47:30.513906 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_incremental [607..608] to osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:30.513918 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 build_incremental [607..608] 2013-04-03 10:47:30.513962 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 _reply_map 607 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:30.513982 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_latest to osd.3 10.252.24.88:6804/18376 start 607 2013-04-03 10:47:30.513988 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_incremental [607..608] to osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:30.513999 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 build_incremental [607..608] 2013-04-03 10:47:30.514040 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 _reply_map 607 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:30.514058 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_latest to osd.1 10.252.24.25:6805/3572 start 607 2013-04-03 10:47:30.514064 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_incremental [607..608] to osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:30.514075 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 build_incremental [607..608] 2013-04-03 10:47:30.514117 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 _reply_map 607 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:30.514135 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_latest to osd.2 10.252.24.30:6801/5656 start 607 2013-04-03 10:47:30.514141 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_incremental [607..608] to osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:30.514153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 build_incremental [607..608] 2013-04-03 10:47:30.514199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 update_logger 2013-04-03 10:47:30.514206 7fcdffcdb700 0 log [INF] : osdmap e608: 5 osds: 5 up, 5 in 2013-04-03 10:47:30.514267 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos 2013-04-03 10:47:30.514282 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 2013-04-03 10:47:30.514291 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:30.514301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:47:30.514331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:30.514342 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:47:30.514353 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:29.675620 mon.0 10.252.24.25:6789/0 217 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:30.514371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:29.778201 mon.0 10.252.24.25:6789/0 218 : [INF] pgmap v29142: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 9167B/s wr, 1op/s 2013-04-03 10:47:30.515073 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=608}) v2 2013-04-03 10:47:30.515079 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 2013-04-03 10:47:30.515085 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_incremental [608..608] to osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:30.515116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 build_incremental [608..608] 2013-04-03 10:47:30.668963 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 update_from_paxos applying incremental 29143 2013-04-03 10:47:30.668973 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 v29143: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 4466B/s wr, 0op/s 2013-04-03 10:47:30.668993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 send_pg_creates to 0 pgs 2013-04-03 10:47:30.668996 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 update_logger 2013-04-03 10:47:30.669006 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 create_pending v 29144 2013-04-03 10:47:30.669017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 check_osd_map already seen 608 >= 608 2013-04-03 10:47:30.669021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 update_logger 2013-04-03 10:47:30.669032 7fcdffcdb700 0 log [INF] : pgmap v29143: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 4466B/s wr, 0op/s 2013-04-03 10:47:30.669097 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos 2013-04-03 10:47:30.669118 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 2013-04-03 10:47:30.669137 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:30.669186 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos 2013-04-03 10:47:30.669202 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 2013-04-03 10:47:30.669212 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:30.669223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:47:30.669253 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:30.669265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:47:30.669276 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:29.778201 mon.0 10.252.24.25:6789/0 218 : [INF] pgmap v29142: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 9167B/s wr, 1op/s 2013-04-03 10:47:30.669297 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:30.514207 mon.0 10.252.24.25:6789/0 219 : [INF] osdmap e608: 5 osds: 5 up, 5 in 2013-04-03 10:47:30.725605 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos 2013-04-03 10:47:30.725625 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 2013-04-03 10:47:30.725643 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:30.828265 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 encode_pending v394130 2013-04-03 10:47:30.970149 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos 2013-04-03 10:47:30.970177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos version 394130 summary v 394129 2013-04-03 10:47:30.970192 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos latest full 393798 2013-04-03 10:47:30.970210 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos applying incremental log 394130 2013-04-03 10:47:29.675571 mon.0 10.252.24.25:6789/0 216 : [INF] mds.? 10.252.24.88:6800/18040 up:boot 2013-04-03 10:47:30.970239 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos applying incremental log 394130 2013-04-03 10:47:29.675620 mon.0 10.252.24.25:6789/0 217 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:30.970260 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos applying incremental log 394130 2013-04-03 10:47:29.778201 mon.0 10.252.24.25:6789/0 218 : [INF] pgmap v29142: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 9167B/s wr, 1op/s 2013-04-03 10:47:30.970280 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos applying incremental log 394130 2013-04-03 10:47:30.514207 mon.0 10.252.24.25:6789/0 219 : [INF] osdmap e608: 5 osds: 5 up, 5 in 2013-04-03 10:47:30.970314 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 check_subs 2013-04-03 10:47:30.970349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 create_pending v 394131 2013-04-03 10:47:30.970372 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:30.970393 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:30.970415 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:30.970433 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:30.970495 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos 2013-04-03 10:47:30.970511 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos version 394130 summary v 394130 2013-04-03 10:47:30.970529 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:30.995896 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos 2013-04-03 10:47:30.995914 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos version 394130 summary v 394130 2013-04-03 10:47:30.995929 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:31.600863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140496 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:47:32.947414 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:32.947427 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273b600 from 10.252.24.88:6789/0 2013-04-03 10:47:32.947475 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 211 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:47:33.385343 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 check_down_pgs 2013-04-03 10:47:33.385374 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 v29143: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 4466B/s wr, 0op/s 2013-04-03 10:47:33.385410 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:33.385434 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:47:33.385472 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 593 2013-04-03 10:47:33.385505 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos 2013-04-03 10:47:33.385524 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos version 394130 summary v 394130 2013-04-03 10:47:33.385534 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 log 2013-04-03 10:47:33.385551 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:33.385569 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth 2013-04-03 10:47:33.436462 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 2 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:33.848769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 preprocess_query pg_stats(73 pgs tid 1 v 608) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:33.848779 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 prepare_update pg_stats(73 pgs tid 1 v 608) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:33.848789 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 prepare_pg_stats pg_stats(73 pgs tid 1 v 608) v1 from osd.3 2013-04-03 10:47:33.848797 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 got osd.3 osd_stat(1188 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:47:33.899491 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 encode_pending v 29144 2013-04-03 10:47:33.934035 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 update_from_paxos applying incremental 29144 2013-04-03 10:47:33.934218 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 v29144: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:47:33.934247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 send_pg_creates to 0 pgs 2013-04-03 10:47:33.934252 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 update_logger 2013-04-03 10:47:33.934271 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 create_pending v 29145 2013-04-03 10:47:33.934281 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 _updated_stats for osd.3 10.252.24.88:6804/18376 2013-04-03 10:47:33.934311 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 check_osd_map already seen 608 >= 608 2013-04-03 10:47:33.934315 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 update_logger 2013-04-03 10:47:33.934325 7fcdffcdb700 0 log [INF] : pgmap v29144: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:47:33.934391 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos 2013-04-03 10:47:33.934412 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos version 394130 summary v 394130 2013-04-03 10:47:33.934431 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:33.934476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos 2013-04-03 10:47:33.934491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos version 394130 summary v 394130 2013-04-03 10:47:33.934501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:33.934511 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:47:33.934542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:33.934554 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:47:33.934564 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 logging 2013-04-03 10:47:30.669033 mon.0 10.252.24.25:6789/0 220 : [INF] pgmap v29143: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 4466B/s wr, 0op/s 2013-04-03 10:47:33.934585 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 logging 2013-04-03 10:47:33.934326 mon.0 10.252.24.25:6789/0 221 : [INF] pgmap v29144: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:47:33.966871 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos 2013-04-03 10:47:33.966889 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos version 394130 summary v 394130 2013-04-03 10:47:33.966904 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:34.551227 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:34.551240 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 2013-04-03 10:47:34.551292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 preprocess_query pg_stats(0 pgs tid 64 v 608) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:47:34.551301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 prepare_update pg_stats(0 pgs tid 64 v 608) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:47:34.551305 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 prepare_pg_stats pg_stats(0 pgs tid 64 v 608) v1 from osd.4 2013-04-03 10:47:34.551312 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 got osd.4 osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[])) 2013-04-03 10:47:34.644462 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 preprocess_query pg_stats(45 pgs tid 70 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:34.644471 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 prepare_update pg_stats(45 pgs tid 70 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:34.644476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 prepare_pg_stats pg_stats(45 pgs tid 70 v 608) v1 from osd.1 2013-04-03 10:47:34.644482 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) 2013-04-03 10:47:34.869773 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 preprocess_query pg_stats(51 pgs tid 413 v 608) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:34.869782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 prepare_update pg_stats(51 pgs tid 413 v 608) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:34.869786 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 prepare_pg_stats pg_stats(51 pgs tid 413 v 608) v1 from osd.0 2013-04-03 10:47:34.869792 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 got osd.0 osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1141 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:47:34.933200 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 encode_pending v394131 2013-04-03 10:47:34.934766 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 encode_pending v 29145 2013-04-03 10:47:34.989973 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos 2013-04-03 10:47:34.990001 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos version 394131 summary v 394130 2013-04-03 10:47:34.990017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos latest full 393798 2013-04-03 10:47:34.990035 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos applying incremental log 394131 2013-04-03 10:47:30.669033 mon.0 10.252.24.25:6789/0 220 : [INF] pgmap v29143: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 4466B/s wr, 0op/s 2013-04-03 10:47:34.990067 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos applying incremental log 394131 2013-04-03 10:47:33.934326 mon.0 10.252.24.25:6789/0 221 : [INF] pgmap v29144: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:47:34.990662 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 check_subs 2013-04-03 10:47:34.990704 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 create_pending v 394132 2013-04-03 10:47:34.990728 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:35.029524 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 update_from_paxos applying incremental 29145 2013-04-03 10:47:35.029757 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 v29145: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 926B/s wr, 0op/s 2013-04-03 10:47:35.029790 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 send_pg_creates to 0 pgs 2013-04-03 10:47:35.029795 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 update_logger 2013-04-03 10:47:35.029814 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 create_pending v 29146 2013-04-03 10:47:35.029823 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 _updated_stats for osd.4 10.252.24.32:6800/13473 2013-04-03 10:47:35.029839 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:35.029864 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:47:35.029887 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 check_osd_map already seen 608 >= 608 2013-04-03 10:47:35.029891 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 update_logger 2013-04-03 10:47:35.029902 7fcdffcdb700 0 log [INF] : pgmap v29145: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 926B/s wr, 0op/s 2013-04-03 10:47:35.029961 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos 2013-04-03 10:47:35.029980 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos version 394131 summary v 394131 2013-04-03 10:47:35.029999 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:35.030043 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos 2013-04-03 10:47:35.030059 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos version 394131 summary v 394131 2013-04-03 10:47:35.030068 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:35.030079 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:47:35.030098 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:35.030109 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:47:35.030119 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 logging 2013-04-03 10:47:35.029903 mon.0 10.252.24.25:6789/0 222 : [INF] pgmap v29145: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 926B/s wr, 0op/s 2013-04-03 10:47:35.062050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos 2013-04-03 10:47:35.062068 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos version 394131 summary v 394131 2013-04-03 10:47:35.062083 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:35.297112 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:35.297121 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:47:35.297154 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 preprocess_query pg_stats(71 pgs tid 52 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:35.297162 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 prepare_update pg_stats(71 pgs tid 52 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:35.297166 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 prepare_pg_stats pg_stats(71 pgs tid 52 v 608) v1 from osd.2 2013-04-03 10:47:35.297172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 got osd.2 osd_stat(1182 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:47:35.600990 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140497 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:47:36.028507 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 encode_pending v394132 2013-04-03 10:47:36.029905 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 encode_pending v 29146 2013-04-03 10:47:36.053357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos 2013-04-03 10:47:36.053385 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos version 394132 summary v 394131 2013-04-03 10:47:36.053400 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos latest full 393798 2013-04-03 10:47:36.053419 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos applying incremental log 394132 2013-04-03 10:47:35.029903 mon.0 10.252.24.25:6789/0 222 : [INF] pgmap v29145: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 926B/s wr, 0op/s 2013-04-03 10:47:36.053464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 check_subs 2013-04-03 10:47:36.053500 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 create_pending v 394133 2013-04-03 10:47:36.053523 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:36.091497 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 update_from_paxos applying incremental 29146 2013-04-03 10:47:36.091681 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 v29146: 240 pgs: 240 active+clean; 256 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 1922B/s wr, 0op/s 2013-04-03 10:47:36.091710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 send_pg_creates to 0 pgs 2013-04-03 10:47:36.091714 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 update_logger 2013-04-03 10:47:36.091729 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 create_pending v 29147 2013-04-03 10:47:36.091739 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:36.091776 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 check_osd_map already seen 608 >= 608 2013-04-03 10:47:36.091780 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 update_logger 2013-04-03 10:47:36.091791 7fcdffcdb700 0 log [INF] : pgmap v29146: 240 pgs: 240 active+clean; 256 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 1922B/s wr, 0op/s 2013-04-03 10:47:36.091851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos 2013-04-03 10:47:36.091869 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos version 394132 summary v 394132 2013-04-03 10:47:36.091889 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:36.091932 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos 2013-04-03 10:47:36.091947 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos version 394132 summary v 394132 2013-04-03 10:47:36.091957 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:36.091969 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:47:36.091989 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:36.092003 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:47:36.092013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 logging 2013-04-03 10:47:36.091792 mon.0 10.252.24.25:6789/0 223 : [INF] pgmap v29146: 240 pgs: 240 active+clean; 256 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 1922B/s wr, 0op/s 2013-04-03 10:47:36.124056 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos 2013-04-03 10:47:36.124073 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos version 394132 summary v 394132 2013-04-03 10:47:36.124088 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:36.947551 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:36.947561 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ce2340 from 10.252.24.88:6789/0 2013-04-03 10:47:36.947618 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 212 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:47:37.090505 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 encode_pending v394133 2013-04-03 10:47:37.115476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos 2013-04-03 10:47:37.115504 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394132 2013-04-03 10:47:37.115518 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos latest full 393798 2013-04-03 10:47:37.115538 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos applying incremental log 394133 2013-04-03 10:47:36.091792 mon.0 10.252.24.25:6789/0 223 : [INF] pgmap v29146: 240 pgs: 240 active+clean; 256 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 1922B/s wr, 0op/s 2013-04-03 10:47:37.115585 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 check_subs 2013-04-03 10:47:37.115621 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 create_pending v 394134 2013-04-03 10:47:37.115644 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:37.115707 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos 2013-04-03 10:47:37.115724 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 2013-04-03 10:47:37.115743 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:37.127030 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos 2013-04-03 10:47:37.127047 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 2013-04-03 10:47:37.127062 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:37.436641 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 3 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:38.385720 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 check_down_pgs 2013-04-03 10:47:38.385749 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 v29146: 240 pgs: 240 active+clean; 256 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 1922B/s wr, 0op/s 2013-04-03 10:47:38.385779 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:38.385804 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:47:38.385842 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 2013-04-03 10:47:38.385874 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos 2013-04-03 10:47:38.385893 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 2013-04-03 10:47:38.385903 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 log 2013-04-03 10:47:38.385920 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:38.385939 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth 2013-04-03 10:47:39.601113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140498 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:47:40.116444 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos 2013-04-03 10:47:40.116465 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 2013-04-03 10:47:40.116483 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:40.947631 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:40.947642 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ce1580 from 10.252.24.88:6789/0 2013-04-03 10:47:40.947699 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 213 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:47:41.436704 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 4 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:43.116471 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos 2013-04-03 10:47:43.116492 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 2013-04-03 10:47:43.116511 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:43.386096 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 v29146: 240 pgs: 240 active+clean; 256 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 1922B/s wr, 0op/s 2013-04-03 10:47:43.386139 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:43.386163 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:47:43.386206 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 2013-04-03 10:47:43.386238 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos 2013-04-03 10:47:43.386257 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 2013-04-03 10:47:43.386267 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 log 2013-04-03 10:47:43.386286 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:43.386306 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth 2013-04-03 10:47:43.601224 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140499 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:47:44.947820 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:44.947831 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a848c0 from 10.252.24.88:6789/0 2013-04-03 10:47:44.947887 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 214 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:47:45.293316 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:45.293327 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 2013-04-03 10:47:45.293382 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 preprocess_query pg_stats(1 pgs tid 53 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:45.293390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 prepare_update pg_stats(1 pgs tid 53 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:45.293397 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 prepare_pg_stats pg_stats(1 pgs tid 53 v 608) v1 from osd.2 2013-04-03 10:47:45.293404 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 got osd.2 osd_stat(1181 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1182 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:47:45.343587 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 encode_pending v 29147 2013-04-03 10:47:45.437121 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 5 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:45.437927 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 update_from_paxos applying incremental 29147 2013-04-03 10:47:45.437950 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 v29147: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 7870B/s wr, 0op/s 2013-04-03 10:47:45.437971 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 send_pg_creates to 0 pgs 2013-04-03 10:47:45.438251 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 update_logger 2013-04-03 10:47:45.438269 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 create_pending v 29148 2013-04-03 10:47:45.438281 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:47:45.438300 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 check_osd_map already seen 608 >= 608 2013-04-03 10:47:45.438304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 update_logger 2013-04-03 10:47:45.438331 7fcdffcdb700 0 log [INF] : pgmap v29147: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 7870B/s wr, 0op/s 2013-04-03 10:47:45.438404 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos 2013-04-03 10:47:45.438424 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 2013-04-03 10:47:45.438443 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:45.438492 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos 2013-04-03 10:47:45.438508 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 2013-04-03 10:47:45.438517 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:45.438531 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:47:45.438555 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:45.438568 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:47:45.438578 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 logging 2013-04-03 10:47:45.438334 mon.0 10.252.24.25:6789/0 224 : [INF] pgmap v29147: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 7870B/s wr, 0op/s 2013-04-03 10:47:45.461668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos 2013-04-03 10:47:45.461690 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 2013-04-03 10:47:45.461707 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:46.437683 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 encode_pending v394134 2013-04-03 10:47:46.461233 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos 2013-04-03 10:47:46.461262 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394133 2013-04-03 10:47:46.461277 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos latest full 393798 2013-04-03 10:47:46.461294 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos applying incremental log 394134 2013-04-03 10:47:45.438334 mon.0 10.252.24.25:6789/0 224 : [INF] pgmap v29147: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 7870B/s wr, 0op/s 2013-04-03 10:47:46.461345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 check_subs 2013-04-03 10:47:46.461381 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 create_pending v 394135 2013-04-03 10:47:46.461404 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:46.461470 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos 2013-04-03 10:47:46.461486 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 2013-04-03 10:47:46.461505 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:46.474165 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos 2013-04-03 10:47:46.474182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 2013-04-03 10:47:46.474200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:47.601359 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140500 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:47:48.386426 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 check_down_pgs 2013-04-03 10:47:48.386457 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 v29147: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 7870B/s wr, 0op/s 2013-04-03 10:47:48.386486 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:48.386510 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:47:48.386549 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 2013-04-03 10:47:48.386580 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos 2013-04-03 10:47:48.386600 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 2013-04-03 10:47:48.386610 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 log 2013-04-03 10:47:48.386628 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:48.386652 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth 2013-04-03 10:47:48.948058 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:48.948069 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a528c0 from 10.252.24.88:6789/0 2013-04-03 10:47:48.948125 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 215 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:47:49.436964 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 6 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:49.462114 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos 2013-04-03 10:47:49.462135 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 2013-04-03 10:47:49.462153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:51.601454 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140501 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:47:52.462062 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos 2013-04-03 10:47:52.462083 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 2013-04-03 10:47:52.462102 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:52.948069 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:52.948080 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2f79580 from 10.252.24.88:6789/0 2013-04-03 10:47:52.948137 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 216 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:47:53.386779 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 v29147: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 7870B/s wr, 0op/s 2013-04-03 10:47:53.386822 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:53.386846 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:47:53.386892 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 2013-04-03 10:47:53.386924 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos 2013-04-03 10:47:53.386944 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 2013-04-03 10:47:53.386953 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 log 2013-04-03 10:47:53.386969 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:53.386989 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth 2013-04-03 10:47:53.437002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 7 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:54.646533 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 preprocess_query pg_stats(1 pgs tid 71 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:54.646543 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 prepare_update pg_stats(1 pgs tid 71 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:54.646547 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 prepare_pg_stats pg_stats(1 pgs tid 71 v 608) v1 from osd.1 2013-04-03 10:47:54.646554 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) 2013-04-03 10:47:54.696755 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 encode_pending v 29148 2013-04-03 10:47:54.726528 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 update_from_paxos applying incremental 29148 2013-04-03 10:47:54.726548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 v29148: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4607B/s wr, 0op/s 2013-04-03 10:47:54.726567 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 send_pg_creates to 0 pgs 2013-04-03 10:47:54.726569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 update_logger 2013-04-03 10:47:54.726579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 create_pending v 29149 2013-04-03 10:47:54.726589 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:47:54.726605 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 check_osd_map already seen 608 >= 608 2013-04-03 10:47:54.726608 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 update_logger 2013-04-03 10:47:54.726618 7fcdffcdb700 0 log [INF] : pgmap v29148: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4607B/s wr, 0op/s 2013-04-03 10:47:54.726680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos 2013-04-03 10:47:54.726701 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 2013-04-03 10:47:54.726719 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:54.726766 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos 2013-04-03 10:47:54.726781 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 2013-04-03 10:47:54.726789 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:54.726801 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:47:54.726825 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:47:54.726837 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:47:54.726847 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 logging 2013-04-03 10:47:54.726619 mon.0 10.252.24.25:6789/0 225 : [INF] pgmap v29148: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4607B/s wr, 0op/s 2013-04-03 10:47:54.738170 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos 2013-04-03 10:47:54.738191 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 2013-04-03 10:47:54.738209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:55.601593 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140502 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:47:55.726262 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 encode_pending v394135 2013-04-03 10:47:55.762971 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos 2013-04-03 10:47:55.763009 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos version 394135 summary v 394134 2013-04-03 10:47:55.763027 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos latest full 393798 2013-04-03 10:47:55.763046 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos applying incremental log 394135 2013-04-03 10:47:54.726619 mon.0 10.252.24.25:6789/0 225 : [INF] pgmap v29148: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4607B/s wr, 0op/s 2013-04-03 10:47:55.763098 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 check_subs 2013-04-03 10:47:55.763135 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 create_pending v 394136 2013-04-03 10:47:55.763158 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:47:55.763224 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos 2013-04-03 10:47:55.763241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos version 394135 summary v 394135 2013-04-03 10:47:55.763261 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:55.775769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos 2013-04-03 10:47:55.775786 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos version 394135 summary v 394135 2013-04-03 10:47:55.775801 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:56.948254 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:47:56.948265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc4000 from 10.252.24.88:6789/0 2013-04-03 10:47:56.948323 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 217 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:47:57.437186 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 8 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:47:58.387109 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 check_down_pgs 2013-04-03 10:47:58.387141 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 v29148: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4607B/s wr, 0op/s 2013-04-03 10:47:58.387172 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:47:58.387196 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:47:58.387235 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 2013-04-03 10:47:58.387266 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos 2013-04-03 10:47:58.387286 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos version 394135 summary v 394135 2013-04-03 10:47:58.387296 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 log 2013-04-03 10:47:58.387327 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:58.387347 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth 2013-04-03 10:47:58.763840 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos 2013-04-03 10:47:58.763862 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos version 394135 summary v 394135 2013-04-03 10:47:58.763879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:47:59.107977 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) 2013-04-03 10:47:59.108026 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader 2013-04-03 10:47:59.601685 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140503 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:48:00.948258 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:00.948268 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc42c0 from 10.252.24.88:6789/0 2013-04-03 10:48:00.948326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 218 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:48:01.437312 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 9 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:48:01.763976 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos 2013-04-03 10:48:01.763996 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos version 394135 summary v 394135 2013-04-03 10:48:01.764014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:03.387472 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29148 v29148: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4607B/s wr, 0op/s 2013-04-03 10:48:03.387516 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:03.387539 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:48:03.387584 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:48:03.387626 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos 2013-04-03 10:48:03.387644 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos version 394135 summary v 394135 2013-04-03 10:48:03.387652 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394135 log 2013-04-03 10:48:03.387669 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:03.387688 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:48:03.602333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140504 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:48:03.849295 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29148 preprocess_query pg_stats(1 pgs tid 2 v 608) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:48:03.849321 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29148 prepare_update pg_stats(1 pgs tid 2 v 608) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:48:03.849325 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29148 prepare_pg_stats pg_stats(1 pgs tid 2 v 608) v1 from osd.3 2013-04-03 10:48:03.849332 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29148 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1188 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:48:03.899521 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29148 encode_pending v 29149 2013-04-03 10:48:04.295191 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29148 update_from_paxos applying incremental 29149 2013-04-03 10:48:04.295213 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 v29149: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 275B/s wr, 0op/s 2013-04-03 10:48:04.295232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 send_pg_creates to 0 pgs 2013-04-03 10:48:04.295238 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 update_logger 2013-04-03 10:48:04.295249 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 create_pending v 29150 2013-04-03 10:48:04.295258 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29149 _updated_stats for osd.3 10.252.24.88:6804/18376 2013-04-03 10:48:04.295280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 check_osd_map already seen 608 >= 608 2013-04-03 10:48:04.295283 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 update_logger 2013-04-03 10:48:04.295293 7fcdffcdb700 0 log [INF] : pgmap v29149: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 275B/s wr, 0op/s 2013-04-03 10:48:04.295354 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos 2013-04-03 10:48:04.295374 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos version 394135 summary v 394135 2013-04-03 10:48:04.295393 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:04.295437 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos 2013-04-03 10:48:04.295452 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos version 394135 summary v 394135 2013-04-03 10:48:04.295460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:04.295470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:48:04.295497 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:04.295510 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:48:04.295520 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 logging 2013-04-03 10:48:04.295294 mon.0 10.252.24.25:6789/0 226 : [INF] pgmap v29149: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 275B/s wr, 0op/s 2013-04-03 10:48:04.461144 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos 2013-04-03 10:48:04.461164 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos version 394135 summary v 394135 2013-04-03 10:48:04.461182 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:04.948382 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:04.948393 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4840 from 10.252.24.88:6789/0 2013-04-03 10:48:04.948967 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 219 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:48:05.294347 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394135 encode_pending v394136 2013-04-03 10:48:05.295616 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:05.295627 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x33e4b40 from 10.252.24.88:6789/0 2013-04-03 10:48:05.295664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 preprocess_query pg_stats(1 pgs tid 54 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:48:05.295688 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 prepare_update pg_stats(1 pgs tid 54 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:48:05.295694 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 prepare_pg_stats pg_stats(1 pgs tid 54 v 608) v1 from osd.2 2013-04-03 10:48:05.295700 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1181 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:48:05.319197 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos 2013-04-03 10:48:05.319232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos version 394136 summary v 394135 2013-04-03 10:48:05.319247 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos latest full 393798 2013-04-03 10:48:05.319264 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394136 update_from_paxos applying incremental log 394136 2013-04-03 10:48:04.295294 mon.0 10.252.24.25:6789/0 226 : [INF] pgmap v29149: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 275B/s wr, 0op/s 2013-04-03 10:48:05.319313 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 check_subs 2013-04-03 10:48:05.319349 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 create_pending v 394137 2013-04-03 10:48:05.319372 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394136 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:05.319433 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos 2013-04-03 10:48:05.319449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos version 394136 summary v 394136 2013-04-03 10:48:05.319466 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:05.342873 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos 2013-04-03 10:48:05.342893 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos version 394136 summary v 394136 2013-04-03 10:48:05.342908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:05.345844 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29149 encode_pending v 29150 2013-04-03 10:48:05.356378 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29149 update_from_paxos applying incremental 29150 2013-04-03 10:48:05.356397 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 v29150: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2403B/s wr, 0op/s 2013-04-03 10:48:05.356416 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 send_pg_creates to 0 pgs 2013-04-03 10:48:05.356419 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 update_logger 2013-04-03 10:48:05.356428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 create_pending v 29151 2013-04-03 10:48:05.356437 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29150 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:48:05.356453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 check_osd_map already seen 608 >= 608 2013-04-03 10:48:05.356455 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 update_logger 2013-04-03 10:48:05.356465 7fcdffcdb700 0 log [INF] : pgmap v29150: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2403B/s wr, 0op/s 2013-04-03 10:48:05.356526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos 2013-04-03 10:48:05.356544 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos version 394136 summary v 394136 2013-04-03 10:48:05.356560 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:05.356613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos 2013-04-03 10:48:05.356628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos version 394136 summary v 394136 2013-04-03 10:48:05.356636 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:05.356648 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:48:05.356669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:05.356681 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:48:05.356690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 logging 2013-04-03 10:48:05.356466 mon.0 10.252.24.25:6789/0 227 : [INF] pgmap v29150: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2403B/s wr, 0op/s 2013-04-03 10:48:05.368093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos 2013-04-03 10:48:05.368110 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos version 394136 summary v 394136 2013-04-03 10:48:05.368125 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:05.437414 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 10 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:48:06.356177 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394136 encode_pending v394137 2013-04-03 10:48:06.384703 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos 2013-04-03 10:48:06.384733 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394136 2013-04-03 10:48:06.384747 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos latest full 393798 2013-04-03 10:48:06.384764 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394137 update_from_paxos applying incremental log 394137 2013-04-03 10:48:05.356466 mon.0 10.252.24.25:6789/0 227 : [INF] pgmap v29150: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2403B/s wr, 0op/s 2013-04-03 10:48:06.384814 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 check_subs 2013-04-03 10:48:06.384850 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 create_pending v 394138 2013-04-03 10:48:06.384872 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394137 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:06.384937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos 2013-04-03 10:48:06.384953 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394137 2013-04-03 10:48:06.384971 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:06.409603 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos 2013-04-03 10:48:06.409620 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394137 2013-04-03 10:48:06.409639 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:07.601911 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140505 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:48:08.387817 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29150 check_down_pgs 2013-04-03 10:48:08.387852 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29150 v29150: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2403B/s wr, 0op/s 2013-04-03 10:48:08.387881 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:08.387913 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:48:08.387952 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:48:08.387983 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos 2013-04-03 10:48:08.388002 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394137 2013-04-03 10:48:08.388012 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394137 log 2013-04-03 10:48:08.388029 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:08.388048 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:48:08.948585 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:08.948600 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4340 from 10.252.24.88:6789/0 2013-04-03 10:48:08.948660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 220 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:48:09.385620 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos 2013-04-03 10:48:09.385940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394137 2013-04-03 10:48:09.385965 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:09.437513 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 11 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:48:09.648204 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 preprocess_query pg_stats(1 pgs tid 72 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:48:09.648229 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 prepare_update pg_stats(1 pgs tid 72 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:48:09.648237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 prepare_pg_stats pg_stats(1 pgs tid 72 v 608) v1 from osd.1 2013-04-03 10:48:09.648243 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) 2013-04-03 10:48:09.698432 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29150 encode_pending v 29151 2013-04-03 10:48:09.740901 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29150 update_from_paxos applying incremental 29151 2013-04-03 10:48:09.740922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 v29151: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 4944B/s wr, 0op/s 2013-04-03 10:48:09.740941 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 send_pg_creates to 0 pgs 2013-04-03 10:48:09.740943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 update_logger 2013-04-03 10:48:09.740953 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 create_pending v 29152 2013-04-03 10:48:09.740962 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29151 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:48:09.740979 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 check_osd_map already seen 608 >= 608 2013-04-03 10:48:09.740981 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 update_logger 2013-04-03 10:48:09.740991 7fcdffcdb700 0 log [INF] : pgmap v29151: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 4944B/s wr, 0op/s 2013-04-03 10:48:09.741102 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos 2013-04-03 10:48:09.741123 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394137 2013-04-03 10:48:09.741142 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:09.741187 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos 2013-04-03 10:48:09.741202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394137 2013-04-03 10:48:09.741211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:09.741221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:48:09.741244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:09.741255 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:48:09.741265 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 logging 2013-04-03 10:48:09.740992 mon.0 10.252.24.25:6789/0 228 : [INF] pgmap v29151: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 4944B/s wr, 0op/s 2013-04-03 10:48:09.753159 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos 2013-04-03 10:48:09.753176 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394137 2013-04-03 10:48:09.753191 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:10.740615 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394137 encode_pending v394138 2013-04-03 10:48:10.768145 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos 2013-04-03 10:48:10.768187 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394137 2013-04-03 10:48:10.768201 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos latest full 393798 2013-04-03 10:48:10.768219 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394138 update_from_paxos applying incremental log 394138 2013-04-03 10:48:09.740992 mon.0 10.252.24.25:6789/0 228 : [INF] pgmap v29151: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 4944B/s wr, 0op/s 2013-04-03 10:48:10.768267 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 check_subs 2013-04-03 10:48:10.768305 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 create_pending v 394139 2013-04-03 10:48:10.768327 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394138 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:10.768393 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos 2013-04-03 10:48:10.768409 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 2013-04-03 10:48:10.768426 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:10.780610 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos 2013-04-03 10:48:10.780632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 2013-04-03 10:48:10.780649 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:11.602024 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140506 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:48:12.948681 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:12.948691 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x346fb80 from 10.252.24.88:6789/0 2013-04-03 10:48:12.948749 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 221 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:48:13.388176 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29151 check_down_pgs 2013-04-03 10:48:13.388208 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29151 v29151: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 4944B/s wr, 0op/s 2013-04-03 10:48:13.388238 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:13.388261 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:48:13.388299 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:48:13.388332 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos 2013-04-03 10:48:13.388351 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 2013-04-03 10:48:13.388360 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394138 log 2013-04-03 10:48:13.388377 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:13.388396 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:48:13.437630 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 12 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:48:13.769037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos 2013-04-03 10:48:13.769057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 2013-04-03 10:48:13.769075 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:15.602161 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140507 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:48:16.769107 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos 2013-04-03 10:48:16.769128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 2013-04-03 10:48:16.769151 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:16.948798 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:16.948808 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x346f8c0 from 10.252.24.88:6789/0 2013-04-03 10:48:16.948865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 222 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:48:17.437774 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 13 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:48:18.388521 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29151 v29151: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 4944B/s wr, 0op/s 2013-04-03 10:48:18.388564 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:18.388587 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:48:18.388632 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:48:18.388661 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos 2013-04-03 10:48:18.388689 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 2013-04-03 10:48:18.388697 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394138 log 2013-04-03 10:48:18.388714 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:18.388733 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:48:19.602298 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140508 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:48:19.769232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos 2013-04-03 10:48:19.769252 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 2013-04-03 10:48:19.769270 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:20.297037 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:20.297047 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x33e4b40 from 10.252.24.88:6789/0 2013-04-03 10:48:20.297104 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 preprocess_query pg_stats(1 pgs tid 55 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:48:20.297128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 prepare_update pg_stats(1 pgs tid 55 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:48:20.297134 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 prepare_pg_stats pg_stats(1 pgs tid 55 v 608) v1 from osd.2 2013-04-03 10:48:20.297140 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:48:20.347312 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29151 encode_pending v 29152 2013-04-03 10:48:20.394487 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29151 update_from_paxos applying incremental 29152 2013-04-03 10:48:20.394508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 v29152: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:48:20.394528 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 send_pg_creates to 0 pgs 2013-04-03 10:48:20.394530 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 update_logger 2013-04-03 10:48:20.394540 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 create_pending v 29153 2013-04-03 10:48:20.394549 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29152 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:48:20.394566 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 check_osd_map already seen 608 >= 608 2013-04-03 10:48:20.394568 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 update_logger 2013-04-03 10:48:20.394578 7fcdffcdb700 0 log [INF] : pgmap v29152: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:48:20.394652 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos 2013-04-03 10:48:20.394672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 2013-04-03 10:48:20.394690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:20.394733 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos 2013-04-03 10:48:20.394748 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 2013-04-03 10:48:20.394757 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:20.394768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:48:20.394790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:20.394801 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:48:20.394810 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 logging 2013-04-03 10:48:20.394580 mon.0 10.252.24.25:6789/0 229 : [INF] pgmap v29152: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:48:20.407229 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos 2013-04-03 10:48:20.407249 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 2013-04-03 10:48:20.407266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:20.948934 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:20.948947 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x346f340 from 10.252.24.88:6789/0 2013-04-03 10:48:20.949002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 223 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:48:21.394291 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394138 encode_pending v394139 2013-04-03 10:48:21.423387 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos 2013-04-03 10:48:21.423426 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394138 2013-04-03 10:48:21.423441 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos latest full 393798 2013-04-03 10:48:21.423458 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394139 update_from_paxos applying incremental log 394139 2013-04-03 10:48:20.394580 mon.0 10.252.24.25:6789/0 229 : [INF] pgmap v29152: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:48:21.423508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 check_subs 2013-04-03 10:48:21.423544 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 create_pending v 394140 2013-04-03 10:48:21.423566 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394139 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:21.423633 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos 2013-04-03 10:48:21.423649 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 2013-04-03 10:48:21.423666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:21.434355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos 2013-04-03 10:48:21.434373 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 2013-04-03 10:48:21.434388 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:21.437724 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 14 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:48:23.388855 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29152 check_down_pgs 2013-04-03 10:48:23.388887 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29152 v29152: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:48:23.388917 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:23.388940 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:48:23.388980 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:48:23.389009 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos 2013-04-03 10:48:23.389032 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 2013-04-03 10:48:23.389040 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394139 log 2013-04-03 10:48:23.389057 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:23.389076 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:48:23.602432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140509 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:48:24.424277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos 2013-04-03 10:48:24.424298 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 2013-04-03 10:48:24.424315 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:24.948943 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:24.948953 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x346f080 from 10.252.24.88:6789/0 2013-04-03 10:48:24.949011 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 224 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:48:25.437937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 15 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:48:27.424360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos 2013-04-03 10:48:27.424380 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 2013-04-03 10:48:27.424398 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:27.602548 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140510 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:48:28.389199 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29152 v29152: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:48:28.389241 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:28.389264 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:48:28.389307 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:48:28.389347 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos 2013-04-03 10:48:28.389366 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 2013-04-03 10:48:28.389375 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394139 log 2013-04-03 10:48:28.389393 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:28.389411 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:48:28.852183 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 preprocess_query pg_stats(1 pgs tid 3 v 608) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:48:28.852209 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 prepare_update pg_stats(1 pgs tid 3 v 608) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:48:28.852214 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 prepare_pg_stats pg_stats(1 pgs tid 3 v 608) v1 from osd.3 2013-04-03 10:48:28.852221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:48:28.902381 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29152 encode_pending v 29153 2013-04-03 10:48:28.942515 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29152 update_from_paxos applying incremental 29153 2013-04-03 10:48:28.942536 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29153 v29153: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 906B/s wr, 0op/s 2013-04-03 10:48:28.942555 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29153 send_pg_creates to 0 pgs 2013-04-03 10:48:28.942557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29153 update_logger 2013-04-03 10:48:28.942567 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29153 create_pending v 29154 2013-04-03 10:48:28.942576 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29153 _updated_stats for osd.3 10.252.24.88:6804/18376 2013-04-03 10:48:28.942596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29153 check_osd_map already seen 608 >= 608 2013-04-03 10:48:28.942599 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29153 update_logger 2013-04-03 10:48:28.942609 7fcdffcdb700 0 log [INF] : pgmap v29153: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 906B/s wr, 0op/s 2013-04-03 10:48:28.942672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos 2013-04-03 10:48:28.942691 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 2013-04-03 10:48:28.942708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:28.942752 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos 2013-04-03 10:48:28.942768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 2013-04-03 10:48:28.942776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:28.942787 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:48:28.942811 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:28.942822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:48:28.942832 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 logging 2013-04-03 10:48:28.942611 mon.0 10.252.24.25:6789/0 230 : [INF] pgmap v29153: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 906B/s wr, 0op/s 2013-04-03 10:48:28.952596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos 2013-04-03 10:48:28.952613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 2013-04-03 10:48:28.952628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:28.952833 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:28.952840 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x346edc0 from 10.252.24.88:6789/0 2013-04-03 10:48:28.952866 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 225 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:48:29.164008 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:48:29.164015 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 2013-04-03 10:48:29.164025 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 2013-04-03 10:48:29.438012 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 16 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:48:29.650491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29153 preprocess_query pg_stats(1 pgs tid 73 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:48:29.650519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29153 prepare_update pg_stats(1 pgs tid 73 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:48:29.650525 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29153 prepare_pg_stats pg_stats(1 pgs tid 73 v 608) v1 from osd.1 2013-04-03 10:48:29.650532 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29153 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) 2013-04-03 10:48:29.932704 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29153 encode_pending v 29154 2013-04-03 10:48:29.933813 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394139 encode_pending v394140 2013-04-03 10:48:30.013339 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29153 update_from_paxos applying incremental 29154 2013-04-03 10:48:30.013361 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 v29154: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s 2013-04-03 10:48:30.013381 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 send_pg_creates to 0 pgs 2013-04-03 10:48:30.013384 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 update_logger 2013-04-03 10:48:30.013395 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 create_pending v 29155 2013-04-03 10:48:30.013404 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:48:30.013422 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 check_osd_map already seen 608 >= 608 2013-04-03 10:48:30.013425 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 update_logger 2013-04-03 10:48:30.013435 7fcdffcdb700 0 log [INF] : pgmap v29154: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s 2013-04-03 10:48:30.049326 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos 2013-04-03 10:48:30.049365 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos version 394140 summary v 394139 2013-04-03 10:48:30.049380 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos latest full 393798 2013-04-03 10:48:30.049398 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos applying incremental log 394140 2013-04-03 10:48:28.942611 mon.0 10.252.24.25:6789/0 230 : [INF] pgmap v29153: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 906B/s wr, 0op/s 2013-04-03 10:48:30.049449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 check_subs 2013-04-03 10:48:30.049486 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 create_pending v 394141 2013-04-03 10:48:30.049509 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:30.049546 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos 2013-04-03 10:48:30.049561 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos version 394140 summary v 394140 2013-04-03 10:48:30.049571 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:30.049583 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:48:30.049601 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 nothing new 2013-04-03 10:48:30.049661 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos 2013-04-03 10:48:30.049677 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos version 394140 summary v 394140 2013-04-03 10:48:30.049696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:30.060740 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos 2013-04-03 10:48:30.060765 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos version 394140 summary v 394140 2013-04-03 10:48:30.060784 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:30.297851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:30.297862 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x33e4d80 from 10.252.24.88:6789/0 2013-04-03 10:48:30.297917 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 preprocess_query pg_stats(1 pgs tid 56 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:48:30.297941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 prepare_update pg_stats(1 pgs tid 56 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:48:30.297948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 prepare_pg_stats pg_stats(1 pgs tid 56 v 608) v1 from osd.2 2013-04-03 10:48:30.297955 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:48:31.049134 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 encode_pending v 29155 2013-04-03 10:48:31.076808 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 update_from_paxos applying incremental 29155 2013-04-03 10:48:31.076829 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 v29155: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5723B/s wr, 0op/s 2013-04-03 10:48:31.076848 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 send_pg_creates to 0 pgs 2013-04-03 10:48:31.076851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 update_logger 2013-04-03 10:48:31.076865 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 create_pending v 29156 2013-04-03 10:48:31.076874 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:48:31.076892 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 check_osd_map already seen 608 >= 608 2013-04-03 10:48:31.076895 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 update_logger 2013-04-03 10:48:31.076905 7fcdffcdb700 0 log [INF] : pgmap v29155: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5723B/s wr, 0op/s 2013-04-03 10:48:31.077281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos 2013-04-03 10:48:31.077304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos version 394140 summary v 394140 2013-04-03 10:48:31.077322 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:31.077366 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos 2013-04-03 10:48:31.077381 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos version 394140 summary v 394140 2013-04-03 10:48:31.077391 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:31.077402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:48:31.077434 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:31.077447 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:48:31.077457 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 logging 2013-04-03 10:48:30.013436 mon.0 10.252.24.25:6789/0 231 : [INF] pgmap v29154: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s 2013-04-03 10:48:31.077480 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 logging 2013-04-03 10:48:31.076907 mon.0 10.252.24.25:6789/0 232 : [INF] pgmap v29155: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5723B/s wr, 0op/s 2013-04-03 10:48:31.087991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos 2013-04-03 10:48:31.088008 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos version 394140 summary v 394140 2013-04-03 10:48:31.088024 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:31.602692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140511 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:48:32.076466 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 encode_pending v394141 2013-04-03 10:48:32.237016 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos 2013-04-03 10:48:32.237053 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos version 394141 summary v 394140 2013-04-03 10:48:32.237069 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos latest full 393798 2013-04-03 10:48:32.237087 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos applying incremental log 394141 2013-04-03 10:48:30.013436 mon.0 10.252.24.25:6789/0 231 : [INF] pgmap v29154: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s 2013-04-03 10:48:32.237119 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos applying incremental log 394141 2013-04-03 10:48:31.076907 mon.0 10.252.24.25:6789/0 232 : [INF] pgmap v29155: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5723B/s wr, 0op/s 2013-04-03 10:48:32.237163 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 check_subs 2013-04-03 10:48:32.237200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 create_pending v 394142 2013-04-03 10:48:32.237223 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:32.237289 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos 2013-04-03 10:48:32.237305 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos version 394141 summary v 394141 2013-04-03 10:48:32.237325 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:32.249582 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos 2013-04-03 10:48:32.249599 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos version 394141 summary v 394141 2013-04-03 10:48:32.249615 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:32.949564 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:32.949575 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x34ad080 from 10.252.24.88:6789/0 2013-04-03 10:48:32.949634 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 226 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:48:33.389535 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 check_down_pgs 2013-04-03 10:48:33.389569 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 v29155: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5723B/s wr, 0op/s 2013-04-03 10:48:33.389600 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:33.389633 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:48:33.389672 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 2013-04-03 10:48:33.389704 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos 2013-04-03 10:48:33.389724 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos version 394141 summary v 394141 2013-04-03 10:48:33.389734 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 log 2013-04-03 10:48:33.389752 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:33.389771 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth 2013-04-03 10:48:33.438083 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 17 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:48:34.651304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 preprocess_query pg_stats(1 pgs tid 74 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:48:34.651330 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 prepare_update pg_stats(1 pgs tid 74 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:48:34.651335 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 prepare_pg_stats pg_stats(1 pgs tid 74 v 608) v1 from osd.1 2013-04-03 10:48:34.651343 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) 2013-04-03 10:48:34.701523 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 encode_pending v 29156 2013-04-03 10:48:34.728541 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 update_from_paxos applying incremental 29156 2013-04-03 10:48:34.728563 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 v29156: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2576B/s wr, 0op/s 2013-04-03 10:48:34.728582 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 send_pg_creates to 0 pgs 2013-04-03 10:48:34.728586 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 update_logger 2013-04-03 10:48:34.728597 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 create_pending v 29157 2013-04-03 10:48:34.728606 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:48:34.728630 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 check_osd_map already seen 608 >= 608 2013-04-03 10:48:34.728637 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 update_logger 2013-04-03 10:48:34.728649 7fcdffcdb700 0 log [INF] : pgmap v29156: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2576B/s wr, 0op/s 2013-04-03 10:48:34.728715 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos 2013-04-03 10:48:34.728736 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos version 394141 summary v 394141 2013-04-03 10:48:34.728755 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:34.728806 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos 2013-04-03 10:48:34.728822 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos version 394141 summary v 394141 2013-04-03 10:48:34.728830 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:34.728842 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:48:34.728866 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:34.728879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:48:34.728889 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 logging 2013-04-03 10:48:34.728650 mon.0 10.252.24.25:6789/0 233 : [INF] pgmap v29156: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2576B/s wr, 0op/s 2013-04-03 10:48:34.751668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos 2013-04-03 10:48:34.751691 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos version 394141 summary v 394141 2013-04-03 10:48:34.751709 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:35.298378 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:35.298390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x33e4900 from 10.252.24.88:6789/0 2013-04-03 10:48:35.298446 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 preprocess_query pg_stats(1 pgs tid 57 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:48:35.298471 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 prepare_update pg_stats(1 pgs tid 57 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:48:35.298477 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 prepare_pg_stats pg_stats(1 pgs tid 57 v 608) v1 from osd.2 2013-04-03 10:48:35.298485 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:48:35.602817 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140512 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:48:35.728216 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 encode_pending v 29157 2013-04-03 10:48:35.729748 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 encode_pending v394142 2013-04-03 10:48:35.771782 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 update_from_paxos applying incremental 29157 2013-04-03 10:48:35.771806 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 v29157: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1750B/s wr, 0op/s 2013-04-03 10:48:35.771825 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 send_pg_creates to 0 pgs 2013-04-03 10:48:35.771829 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 update_logger 2013-04-03 10:48:35.771843 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 create_pending v 29158 2013-04-03 10:48:35.771852 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:48:35.771878 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 check_osd_map already seen 608 >= 608 2013-04-03 10:48:35.771881 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 update_logger 2013-04-03 10:48:35.771892 7fcdffcdb700 0 log [INF] : pgmap v29157: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1750B/s wr, 0op/s 2013-04-03 10:48:35.800078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos 2013-04-03 10:48:35.800104 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394141 2013-04-03 10:48:35.800123 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos latest full 393798 2013-04-03 10:48:35.800142 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos applying incremental log 394142 2013-04-03 10:48:34.728650 mon.0 10.252.24.25:6789/0 233 : [INF] pgmap v29156: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2576B/s wr, 0op/s 2013-04-03 10:48:35.800187 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 check_subs 2013-04-03 10:48:35.800223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 create_pending v 394143 2013-04-03 10:48:35.800246 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:35.800279 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos 2013-04-03 10:48:35.800294 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 2013-04-03 10:48:35.800304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:35.800315 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:48:35.800335 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 nothing new 2013-04-03 10:48:35.800386 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos 2013-04-03 10:48:35.800402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 2013-04-03 10:48:35.800421 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:35.811841 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos 2013-04-03 10:48:35.811925 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 2013-04-03 10:48:35.811945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:36.949388 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:36.949399 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x34acb00 from 10.252.24.88:6789/0 2013-04-03 10:48:36.949456 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 227 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:48:37.438210 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 18 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:48:38.389923 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 check_down_pgs 2013-04-03 10:48:38.389958 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 v29157: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1750B/s wr, 0op/s 2013-04-03 10:48:38.389988 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:38.390011 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:48:38.390049 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 2013-04-03 10:48:38.390081 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos 2013-04-03 10:48:38.390101 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 2013-04-03 10:48:38.390111 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 log 2013-04-03 10:48:38.390129 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:38.390149 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth 2013-04-03 10:48:38.800899 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos 2013-04-03 10:48:38.800920 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 2013-04-03 10:48:38.800945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:39.602957 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140513 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:48:40.949401 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:40.949414 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x34ac840 from 10.252.24.88:6789/0 2013-04-03 10:48:40.949473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 228 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:48:41.438315 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 19 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:48:41.801021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos 2013-04-03 10:48:41.801043 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 2013-04-03 10:48:41.801061 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:43.390303 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 v29157: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1750B/s wr, 0op/s 2013-04-03 10:48:43.390351 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:43.390375 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:48:43.390418 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 2013-04-03 10:48:43.390450 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos 2013-04-03 10:48:43.390470 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 2013-04-03 10:48:43.390480 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 log 2013-04-03 10:48:43.390497 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:43.390522 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth 2013-04-03 10:48:43.603049 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140514 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:48:44.652215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 preprocess_query pg_stats(1 pgs tid 75 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:48:44.652242 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 prepare_update pg_stats(1 pgs tid 75 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:48:44.652248 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 prepare_pg_stats pg_stats(1 pgs tid 75 v 608) v1 from osd.1 2013-04-03 10:48:44.652254 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) 2013-04-03 10:48:44.702434 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 encode_pending v 29158 2013-04-03 10:48:44.742409 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 update_from_paxos applying incremental 29158 2013-04-03 10:48:44.742430 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s 2013-04-03 10:48:44.742448 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 send_pg_creates to 0 pgs 2013-04-03 10:48:44.742453 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 update_logger 2013-04-03 10:48:44.742464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 create_pending v 29159 2013-04-03 10:48:44.742473 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:48:44.742490 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 check_osd_map already seen 608 >= 608 2013-04-03 10:48:44.742493 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 update_logger 2013-04-03 10:48:44.742504 7fcdffcdb700 0 log [INF] : pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s 2013-04-03 10:48:44.742571 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos 2013-04-03 10:48:44.742592 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 2013-04-03 10:48:44.742614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:44.742660 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos 2013-04-03 10:48:44.742676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 2013-04-03 10:48:44.742686 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:44.742697 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:48:44.742726 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:44.742738 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:48:44.742749 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 logging 2013-04-03 10:48:35.771893 mon.0 10.252.24.25:6789/0 234 : [INF] pgmap v29157: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1750B/s wr, 0op/s 2013-04-03 10:48:44.742770 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 logging 2013-04-03 10:48:44.742505 mon.0 10.252.24.25:6789/0 235 : [INF] pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s 2013-04-03 10:48:44.755211 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos 2013-04-03 10:48:44.755230 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 2013-04-03 10:48:44.755246 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:44.949686 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:44.949699 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x34ac580 from 10.252.24.88:6789/0 2013-04-03 10:48:44.949756 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 229 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:48:45.438455 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 20 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:48:45.742219 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 encode_pending v394143 2013-04-03 10:48:45.778132 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos 2013-04-03 10:48:45.778159 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos version 394143 summary v 394142 2013-04-03 10:48:45.778173 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos latest full 393798 2013-04-03 10:48:45.778190 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos applying incremental log 394143 2013-04-03 10:48:35.771893 mon.0 10.252.24.25:6789/0 234 : [INF] pgmap v29157: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1750B/s wr, 0op/s 2013-04-03 10:48:45.778219 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos applying incremental log 394143 2013-04-03 10:48:44.742505 mon.0 10.252.24.25:6789/0 235 : [INF] pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s 2013-04-03 10:48:45.778262 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 check_subs 2013-04-03 10:48:45.778298 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 create_pending v 394144 2013-04-03 10:48:45.778321 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:45.778388 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos 2013-04-03 10:48:45.778404 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:45.778423 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:45.790405 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos 2013-04-03 10:48:45.790424 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:45.790440 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:47.603195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140515 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:48:48.390644 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 check_down_pgs 2013-04-03 10:48:48.390676 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s 2013-04-03 10:48:48.390706 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:48.390730 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:48:48.390767 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 2013-04-03 10:48:48.390799 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos 2013-04-03 10:48:48.390819 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:48.390828 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 log 2013-04-03 10:48:48.390844 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:48.390864 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth 2013-04-03 10:48:48.418643 7fce0612d700 0 -- 10.252.24.25:6789/0 >> 10.252.24.30:6789/0 pipe(0x1f91280 sd=20 :0 s=1 pgs=0 cs=0 l=0).fault 2013-04-03 10:48:48.419124 7fce0612d700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 ms_get_authorizer for mon 2013-04-03 10:48:48.420486 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 do not have session, making new one 2013-04-03 10:48:48.420491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 ms_dispatch new session MonSession: mon.1 10.252.24.30:6789/0 is open for mon.1 10.252.24.30:6789/0 2013-04-03 10:48:48.420497 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )) e5 setting monitor caps on this connection 2013-04-03 10:48:48.420500 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.420504 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.420571 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.420575 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.420596 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.420599 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.420651 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.420654 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.420714 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.420719 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.420797 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.420803 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.420914 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.420919 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.420988 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.420993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.421016 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.421021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.421068 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.421073 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.421110 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.421115 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.421176 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.421181 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.421209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.421214 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:48.421260 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882 paxos( fc 0 lc 0 )) v4 2013-04-03 10:48:48.421265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_reply mon.1 10.252.24.30:6789/0mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882 paxos( fc 0 lc 0 )) v4 2013-04-03 10:48:48.421270 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 monmap is e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} 2013-04-03 10:48:48.421300 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync mon_sync( heartbeat ) v1 2013-04-03 10:48:48.421303 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_heartbeat mon_sync( heartbeat ) v1 2013-04-03 10:48:48.421316 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_send_heartbeat mon.1 10.252.24.30:6789/0 reply(1) 2013-04-03 10:48:48.781895 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos 2013-04-03 10:48:48.781916 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:48.781934 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos 2013-04-03 10:48:48.954344 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:48.954355 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x34ac2c0 from 10.252.24.88:6789/0 2013-04-03 10:48:48.954413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 230 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:48:49.069193 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync mon_sync( finish ) v1 2013-04-03 10:48:49.069204 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_finish mon_sync( finish ) v1 2013-04-03 10:48:49.069219 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) 2013-04-03 10:48:49.069229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader 2013-04-03 10:48:49.196614 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:49.196622 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:48:49.208756 7fcdffcdb700 5 mon.hotcpc9039@0(leader).elector(354) handle_propose from mon.1 2013-04-03 10:48:49.208760 7fcdffcdb700 5 mon.hotcpc9039@0(leader).elector(354) got propose from old epoch, quorum is 0,2, mon.1 must have just started 2013-04-03 10:48:49.208762 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 reset 2013-04-03 10:48:49.208764 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish 2013-04-03 10:48:49.208771 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 start_election 2013-04-03 10:48:49.208773 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 cancel_probe_timeout (none scheduled) 2013-04-03 10:48:49.208779 7fcdffcdb700 0 log [INF] : mon.hotcpc9039 calling new monitor election 2013-04-03 10:48:49.208797 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(354) start -- can i be leader? 2013-04-03 10:48:49.208834 7fcdffcdb700 1 mon.hotcpc9039@0(electing).elector(354) init, last seen epoch 354 2013-04-03 10:48:49.208839 7fcdffcdb700 10 mon.hotcpc9039@0(electing).elector(354) bump_epoch 354 to 355 2013-04-03 10:48:49.217047 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(355) handle_ack from mon.1 2013-04-03 10:48:49.217051 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(355) so far i have {0=4294967295,1=4294967295} 2013-04-03 10:48:49.253684 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(355) handle_propose from mon.2 2013-04-03 10:48:49.253798 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(355) handle_ack from mon.2 2013-04-03 10:48:49.253801 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(355) so far i have {0=4294967295,1=4294967295,2=4294967295} 2013-04-03 10:48:49.253805 7fcdffcdb700 10 mon.hotcpc9039@0(electing).elector(355) bump_epoch 355 to 356 2013-04-03 10:48:49.254080 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 win_election, epoch 356 quorum is 0,1,2 features are 4294967295 2013-04-03 10:48:49.254089 7fcdffcdb700 0 log [INF] : mon.hotcpc9039@0 won leader election with quorum 0,1,2 2013-04-03 10:48:49.254353 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish 2013-04-03 10:48:49.254357 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 resend_routed_requests 2013-04-03 10:48:49.254359 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 register_cluster_logger - already registered 2013-04-03 10:48:49.254361 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_start 2013-04-03 10:48:49.254362 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round curr 0 2013-04-03 10:48:49.254364 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round new 1 2013-04-03 10:48:49.254365 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck 2013-04-03 10:48:49.254366 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck start timecheck epoch 356 round 1 2013-04-03 10:48:49.254370 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck send time_check( ping e 356 r 1 ) v1 to mon.1 10.252.24.30:6789/0 2013-04-03 10:48:49.254381 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck send time_check( ping e 356 r 1 ) v1 to mon.2 10.252.24.88:6789/0 2013-04-03 10:48:49.254388 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round setting up next event 2013-04-03 10:48:49.266683 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck time_check( pong e 356 r 1 ts 2013-04-03 10:48:49.265712 ) v1 2013-04-03 10:48:49.266688 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader time_check( pong e 356 r 1 ts 2013-04-03 10:48:49.265712 ) v1 2013-04-03 10:48:49.266695 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader from mon.1 10.252.24.30:6789/0 ts 2013-04-03 10:48:49.265712 delta -0.0009799 skew_bound 0 latency 0.012323 2013-04-03 10:48:49.277171 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29158 create_pending v 29159 2013-04-03 10:48:49.277185 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29158 check_osd_map already seen 608 >= 608 2013-04-03 10:48:49.277187 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29158 update_logger 2013-04-03 10:48:49.277223 7fcdffcdb700 0 log [INF] : pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s 2013-04-03 10:48:49.277281 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 create_pending e189 2013-04-03 10:48:49.277302 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:49.277318 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 update_logger 2013-04-03 10:48:49.277324 7fcdffcdb700 0 log [INF] : mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:49.277367 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 create_pending e 609 2013-04-03 10:48:49.277372 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 remove_redundant_pg_temp 2013-04-03 10:48:49.277373 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 remove_down_pg_temp 2013-04-03 10:48:49.277394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 update_logger 2013-04-03 10:48:49.277399 7fcdffcdb700 0 log [INF] : osdmap e608: 5 osds: 5 up, 5 in 2013-04-03 10:48:49.277415 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos 2013-04-03 10:48:49.277439 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:49.277462 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 create_pending v 394144 2013-04-03 10:48:49.278185 7fcdffcdb700 10 mon.hotcpc9039@0(leader).monmap v5 create_pending monmap epoch 6 2013-04-03 10:48:49.278233 7fcdffcdb700 0 log [INF] : monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} 2013-04-03 10:48:49.278257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:49.278282 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 create_pending v 2064 2013-04-03 10:48:49.278301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 AuthMonitor::on_active() 2013-04-03 10:48:49.278325 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos 2013-04-03 10:48:49.278338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:49.278346 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.278356 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:48:49.278379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.278390 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:48:49.278399 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.208780 mon.0 10.252.24.25:6789/0 236 : [INF] mon.hotcpc9039 calling new monitor election 2013-04-03 10:48:49.278440 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos 2013-04-03 10:48:49.278453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:49.278460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.278470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:48:49.278487 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 nothing new 2013-04-03 10:48:49.278524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos 2013-04-03 10:48:49.278538 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:49.278552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:49.278584 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck time_check( pong e 356 r 1 ts 2013-04-03 10:48:49.276057 ) v1 2013-04-03 10:48:49.278588 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader time_check( pong e 356 r 1 ts 2013-04-03 10:48:49.276057 ) v1 2013-04-03 10:48:49.278595 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader from mon.2 10.252.24.88:6789/0 ts 2013-04-03 10:48:49.276057 delta -0.00253391 skew_bound 0 latency 0.024211 2013-04-03 10:48:49.278608 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader got pongs from everybody (3 total) 2013-04-03 10:48:49.278610 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish_round curr 1 2013-04-03 10:48:49.278612 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_report 2013-04-03 10:48:49.278617 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_report send report to mon.1 10.252.24.30:6789/0 2013-04-03 10:48:49.278632 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_report send report to mon.2 10.252.24.88:6789/0 2013-04-03 10:48:49.278668 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos 2013-04-03 10:48:49.278882 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:49.278891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.278903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:48:49.278927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.278937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:48:49.278946 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.208780 mon.0 10.252.24.25:6789/0 236 : [INF] mon.hotcpc9039 calling new monitor election 2013-04-03 10:48:49.278965 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.254090 mon.0 10.252.24.25:6789/0 237 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,1,2 2013-04-03 10:48:49.279024 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos 2013-04-03 10:48:49.279037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:49.279051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:49.279100 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos 2013-04-03 10:48:49.279114 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:49.279128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:49.279162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos 2013-04-03 10:48:49.279177 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:49.279184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.279193 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:48:49.279216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.279226 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:48:49.279235 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.254090 mon.0 10.252.24.25:6789/0 237 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,1,2 2013-04-03 10:48:49.279253 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.277225 mon.0 10.252.24.25:6789/0 238 : [INF] pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s 2013-04-03 10:48:49.279286 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mon.1 10.252.24.30:6789/0 via mon.1 10.252.24.30:6789/0 2013-04-03 10:48:49.279292 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1c15000 from 10.252.24.30:6789/0 2013-04-03 10:48:49.279307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos 2013-04-03 10:48:49.279320 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:49.279328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(1 entries) v1 from mon.1 10.252.24.30:6789/0 2013-04-03 10:48:49.279337 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(1 entries) v1 from mon.1 2013-04-03 10:48:49.279347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_update log(1 entries) v1 from mon.1 10.252.24.30:6789/0 2013-04-03 10:48:49.279357 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_log log(1 entries) v1 from mon.1 2013-04-03 10:48:49.279365 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.196342 mon.1 10.252.24.30:6789/0 1 : [INF] mon.hotcpc9882 calling new monitor election 2013-04-03 10:48:49.279393 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mon.2 10.252.24.88:6789/0 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:49.279399 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x287ea00 from 10.252.24.88:6789/0 2013-04-03 10:48:49.279414 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos 2013-04-03 10:48:49.279427 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:49.279435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(1 entries) v1 from mon.2 10.252.24.88:6789/0 2013-04-03 10:48:49.279445 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(1 entries) v1 from mon.2 2013-04-03 10:48:49.279456 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_update log(1 entries) v1 from mon.2 10.252.24.88:6789/0 2013-04-03 10:48:49.279466 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_log log(1 entries) v1 from mon.2 2013-04-03 10:48:49.279474 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.208409 mon.2 10.252.24.88:6789/0 2 : [INF] mon.hotcpc12533 calling new monitor election 2013-04-03 10:48:49.279514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos 2013-04-03 10:48:49.279527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:49.279535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.279544 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:48:49.279566 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.279576 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:48:49.279584 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.277225 mon.0 10.252.24.25:6789/0 238 : [INF] pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s 2013-04-03 10:48:49.279602 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.277325 mon.0 10.252.24.25:6789/0 239 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:49.279648 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos 2013-04-03 10:48:49.279662 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 2013-04-03 10:48:49.279670 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.279679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:48:49.279702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.279711 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:48:49.279720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.277325 mon.0 10.252.24.25:6789/0 239 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:49.279737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.277400 mon.0 10.252.24.25:6789/0 240 : [INF] osdmap e608: 5 osds: 5 up, 5 in 2013-04-03 10:48:49.328593 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394143 encode_pending v394144 2013-04-03 10:48:49.417635 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos 2013-04-03 10:48:49.417683 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394143 2013-04-03 10:48:49.417697 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos latest full 393798 2013-04-03 10:48:49.417718 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 update_from_paxos applying incremental log 394144 2013-04-03 10:48:49.196342 mon.1 10.252.24.30:6789/0 1 : [INF] mon.hotcpc9882 calling new monitor election 2013-04-03 10:48:49.417744 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 update_from_paxos applying incremental log 394144 2013-04-03 10:48:49.208409 mon.2 10.252.24.88:6789/0 2 : [INF] mon.hotcpc12533 calling new monitor election 2013-04-03 10:48:49.417766 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 update_from_paxos applying incremental log 394144 2013-04-03 10:48:49.208780 mon.0 10.252.24.25:6789/0 236 : [INF] mon.hotcpc9039 calling new monitor election 2013-04-03 10:48:49.417784 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 update_from_paxos applying incremental log 394144 2013-04-03 10:48:49.254090 mon.0 10.252.24.25:6789/0 237 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,1,2 2013-04-03 10:48:49.417804 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 update_from_paxos applying incremental log 394144 2013-04-03 10:48:49.277225 mon.0 10.252.24.25:6789/0 238 : [INF] pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s 2013-04-03 10:48:49.417832 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 update_from_paxos applying incremental log 394144 2013-04-03 10:48:49.277325 mon.0 10.252.24.25:6789/0 239 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:49.417852 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 update_from_paxos applying incremental log 394144 2013-04-03 10:48:49.277400 mon.0 10.252.24.25:6789/0 240 : [INF] osdmap e608: 5 osds: 5 up, 5 in 2013-04-03 10:48:49.417906 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 check_subs 2013-04-03 10:48:49.417945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 create_pending v 394145 2013-04-03 10:48:49.417968 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.417988 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.418007 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.418024 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 _updated_log for mon.1 10.252.24.30:6789/0 2013-04-03 10:48:49.418044 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 _updated_log for mon.2 10.252.24.88:6789/0 2013-04-03 10:48:49.418064 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.418080 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:49.418515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos 2013-04-03 10:48:49.418531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394144 2013-04-03 10:48:49.418548 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:49.418616 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos 2013-04-03 10:48:49.418632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394144 2013-04-03 10:48:49.418647 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:49.424040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos 2013-04-03 10:48:49.424057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394144 2013-04-03 10:48:49.424072 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:49.438488 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 21 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:48:50.299695 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:50.299706 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x33e4900 from 10.252.24.88:6789/0 2013-04-03 10:48:50.299758 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29158 preprocess_query pg_stats(1 pgs tid 58 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:48:50.299782 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29158 prepare_update pg_stats(1 pgs tid 58 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:48:50.299788 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29158 prepare_pg_stats pg_stats(1 pgs tid 58 v 608) v1 from osd.2 2013-04-03 10:48:50.299794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29158 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:48:50.340178 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29158 encode_pending v 29159 2013-04-03 10:48:50.387619 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29158 update_from_paxos applying incremental 29159 2013-04-03 10:48:50.387640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 v29159: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1121B/s wr, 0op/s 2013-04-03 10:48:50.387659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 send_pg_creates to 0 pgs 2013-04-03 10:48:50.387662 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 update_logger 2013-04-03 10:48:50.387672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 create_pending v 29160 2013-04-03 10:48:50.387681 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29159 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:48:50.387698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 check_osd_map already seen 608 >= 608 2013-04-03 10:48:50.387700 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 update_logger 2013-04-03 10:48:50.387710 7fcdffcdb700 0 log [INF] : pgmap v29159: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1121B/s wr, 0op/s 2013-04-03 10:48:50.387794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos 2013-04-03 10:48:50.387814 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394144 2013-04-03 10:48:50.387831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:50.387876 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos 2013-04-03 10:48:50.387891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394144 2013-04-03 10:48:50.387899 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:50.387909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:48:50.387937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:50.387949 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:48:50.387959 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 logging 2013-04-03 10:48:49.278235 mon.0 10.252.24.25:6789/0 241 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} 2013-04-03 10:48:50.387979 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 logging 2013-04-03 10:48:50.387711 mon.0 10.252.24.25:6789/0 242 : [INF] pgmap v29159: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1121B/s wr, 0op/s 2013-04-03 10:48:50.400324 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos 2013-04-03 10:48:50.400342 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394144 2013-04-03 10:48:50.400357 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:50.400973 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos 2013-04-03 10:48:50.400992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394144 2013-04-03 10:48:50.401006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:51.386639 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394144 encode_pending v394145 2013-04-03 10:48:51.421547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos 2013-04-03 10:48:51.421598 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394144 2013-04-03 10:48:51.421613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos latest full 393798 2013-04-03 10:48:51.421638 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394145 update_from_paxos applying incremental log 394145 2013-04-03 10:48:49.278235 mon.0 10.252.24.25:6789/0 241 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} 2013-04-03 10:48:51.421666 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394145 update_from_paxos applying incremental log 394145 2013-04-03 10:48:50.387711 mon.0 10.252.24.25:6789/0 242 : [INF] pgmap v29159: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1121B/s wr, 0op/s 2013-04-03 10:48:51.421710 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 check_subs 2013-04-03 10:48:51.421748 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 create_pending v 394146 2013-04-03 10:48:51.421770 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394145 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:51.421845 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos 2013-04-03 10:48:51.421862 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 2013-04-03 10:48:51.421882 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:51.427483 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos 2013-04-03 10:48:51.427500 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 2013-04-03 10:48:51.427515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:51.446513 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos 2013-04-03 10:48:51.446532 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 2013-04-03 10:48:51.446549 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:51.603355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140516 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:48:52.949767 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:52.949777 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a51580 from 10.252.24.88:6789/0 2013-04-03 10:48:52.949843 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 231 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:48:53.390992 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29159 check_down_pgs 2013-04-03 10:48:53.391025 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29159 v29159: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1121B/s wr, 0op/s 2013-04-03 10:48:53.391054 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:53.391082 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:48:53.391119 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:48:53.391150 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos 2013-04-03 10:48:53.391168 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 2013-04-03 10:48:53.391177 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394145 log 2013-04-03 10:48:53.391194 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:53.391213 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:48:53.438689 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 22 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:48:53.854726 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 preprocess_query pg_stats(1 pgs tid 4 v 608) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:48:53.854751 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 prepare_update pg_stats(1 pgs tid 4 v 608) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:48:53.854756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 prepare_pg_stats pg_stats(1 pgs tid 4 v 608) v1 from osd.3 2013-04-03 10:48:53.854763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:48:53.904949 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29159 encode_pending v 29160 2013-04-03 10:48:53.971674 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29159 update_from_paxos applying incremental 29160 2013-04-03 10:48:53.971695 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 v29160: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1446B/s wr, 0op/s 2013-04-03 10:48:53.971714 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 send_pg_creates to 0 pgs 2013-04-03 10:48:53.971718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 update_logger 2013-04-03 10:48:53.971729 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 create_pending v 29161 2013-04-03 10:48:53.971737 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29160 _updated_stats for osd.3 10.252.24.88:6804/18376 2013-04-03 10:48:53.971753 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 check_osd_map already seen 608 >= 608 2013-04-03 10:48:53.971755 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 update_logger 2013-04-03 10:48:53.971768 7fcdffcdb700 0 log [INF] : pgmap v29160: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1446B/s wr, 0op/s 2013-04-03 10:48:53.971830 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos 2013-04-03 10:48:53.971849 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 2013-04-03 10:48:53.971867 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:53.971909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos 2013-04-03 10:48:53.971925 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 2013-04-03 10:48:53.971933 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:53.971943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:48:53.971966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:53.971979 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:48:53.971989 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 logging 2013-04-03 10:48:53.971769 mon.0 10.252.24.25:6789/0 243 : [INF] pgmap v29160: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1446B/s wr, 0op/s 2013-04-03 10:48:53.973992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos 2013-04-03 10:48:53.974010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 2013-04-03 10:48:53.974026 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:53.977557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos 2013-04-03 10:48:53.978015 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 2013-04-03 10:48:53.978040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:54.653298 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 preprocess_query pg_stats(1 pgs tid 76 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:48:54.653324 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 prepare_update pg_stats(1 pgs tid 76 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:48:54.653329 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 prepare_pg_stats pg_stats(1 pgs tid 76 v 608) v1 from osd.1 2013-04-03 10:48:54.653336 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) 2013-04-03 10:48:54.932622 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394145 encode_pending v394146 2013-04-03 10:48:54.933923 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29160 encode_pending v 29161 2013-04-03 10:48:54.965184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos 2013-04-03 10:48:54.965222 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos version 394146 summary v 394145 2013-04-03 10:48:54.965237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos latest full 393798 2013-04-03 10:48:54.965256 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394146 update_from_paxos applying incremental log 394146 2013-04-03 10:48:53.971769 mon.0 10.252.24.25:6789/0 243 : [INF] pgmap v29160: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1446B/s wr, 0op/s 2013-04-03 10:48:54.965304 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 check_subs 2013-04-03 10:48:54.965342 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 create_pending v 394147 2013-04-03 10:48:54.965365 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394146 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:54.995860 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29160 update_from_paxos applying incremental 29161 2013-04-03 10:48:54.995882 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 v29161: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1114B/s wr, 0op/s 2013-04-03 10:48:54.995900 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 send_pg_creates to 0 pgs 2013-04-03 10:48:54.995903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 update_logger 2013-04-03 10:48:54.995913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 create_pending v 29162 2013-04-03 10:48:54.995922 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29161 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:48:54.995938 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 check_osd_map already seen 608 >= 608 2013-04-03 10:48:54.995940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 update_logger 2013-04-03 10:48:54.995950 7fcdffcdb700 0 log [INF] : pgmap v29161: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1114B/s wr, 0op/s 2013-04-03 10:48:54.996014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos 2013-04-03 10:48:54.996033 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos version 394146 summary v 394146 2013-04-03 10:48:54.996050 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:54.996094 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos 2013-04-03 10:48:54.996109 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos version 394146 summary v 394146 2013-04-03 10:48:54.996118 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:54.996129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:48:54.996152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:48:54.996165 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:48:54.996175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 logging 2013-04-03 10:48:54.995951 mon.0 10.252.24.25:6789/0 244 : [INF] pgmap v29161: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1114B/s wr, 0op/s 2013-04-03 10:48:54.996486 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos 2013-04-03 10:48:54.996502 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos version 394146 summary v 394146 2013-04-03 10:48:54.996518 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:55.004136 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos 2013-04-03 10:48:55.004152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos version 394146 summary v 394146 2013-04-03 10:48:55.004168 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:55.488840 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 2013-04-03 10:48:55.488852 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:48:55.603442 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140517 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:48:55.979577 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394146 encode_pending v394147 2013-04-03 10:48:56.025331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos 2013-04-03 10:48:56.025382 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394146 2013-04-03 10:48:56.025397 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos latest full 393798 2013-04-03 10:48:56.025418 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394147 update_from_paxos applying incremental log 394147 2013-04-03 10:48:54.995951 mon.0 10.252.24.25:6789/0 244 : [INF] pgmap v29161: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1114B/s wr, 0op/s 2013-04-03 10:48:56.025471 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 check_subs 2013-04-03 10:48:56.025507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 create_pending v 394148 2013-04-03 10:48:56.025529 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394147 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:48:56.025597 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos 2013-04-03 10:48:56.025613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 2013-04-03 10:48:56.025632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:56.039913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos 2013-04-03 10:48:56.039931 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 2013-04-03 10:48:56.039947 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:56.050432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos 2013-04-03 10:48:56.050453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 2013-04-03 10:48:56.050469 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:56.949922 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:48:56.949935 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a51840 from 10.252.24.88:6789/0 2013-04-03 10:48:56.949994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 232 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:48:57.438954 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 23 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:48:58.391369 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29161 check_down_pgs 2013-04-03 10:48:58.391397 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29161 v29161: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1114B/s wr, 0op/s 2013-04-03 10:48:58.391425 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:48:58.391447 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:48:58.391481 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:48:58.391513 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos 2013-04-03 10:48:58.391532 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 2013-04-03 10:48:58.391540 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394147 log 2013-04-03 10:48:58.391557 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:58.391575 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:48:59.026176 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos 2013-04-03 10:48:59.026196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 2013-04-03 10:48:59.026214 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:59.026264 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos 2013-04-03 10:48:59.026280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 2013-04-03 10:48:59.026295 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:48:59.603612 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140518 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:49:00.300733 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:00.300745 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34d9480 from 10.252.24.88:6789/0 2013-04-03 10:49:00.300797 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 preprocess_query pg_stats(1 pgs tid 59 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:00.300821 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 prepare_update pg_stats(1 pgs tid 59 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:00.300827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 prepare_pg_stats pg_stats(1 pgs tid 59 v 608) v1 from osd.2 2013-04-03 10:49:00.300833 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:49:00.351018 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29161 encode_pending v 29162 2013-04-03 10:49:00.443488 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29161 update_from_paxos applying incremental 29162 2013-04-03 10:49:00.443507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 v29162: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2541B/s wr, 0op/s 2013-04-03 10:49:00.443526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 send_pg_creates to 0 pgs 2013-04-03 10:49:00.443528 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 update_logger 2013-04-03 10:49:00.443539 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 create_pending v 29163 2013-04-03 10:49:00.443547 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29162 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:00.443567 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 check_osd_map already seen 608 >= 608 2013-04-03 10:49:00.443571 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 update_logger 2013-04-03 10:49:00.443581 7fcdffcdb700 0 log [INF] : pgmap v29162: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2541B/s wr, 0op/s 2013-04-03 10:49:00.443643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos 2013-04-03 10:49:00.443663 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 2013-04-03 10:49:00.443680 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:00.443722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos 2013-04-03 10:49:00.443737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 2013-04-03 10:49:00.443745 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:00.443755 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:49:00.443778 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:00.443790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:49:00.443800 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 logging 2013-04-03 10:49:00.443583 mon.0 10.252.24.25:6789/0 245 : [INF] pgmap v29162: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2541B/s wr, 0op/s 2013-04-03 10:49:00.444134 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos 2013-04-03 10:49:00.444151 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 2013-04-03 10:49:00.444167 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:00.454960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos 2013-04-03 10:49:00.454976 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 2013-04-03 10:49:00.454991 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:00.950087 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:00.950097 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83000 from 10.252.24.88:6789/0 2013-04-03 10:49:00.950152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 233 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:49:01.388765 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394147 encode_pending v394148 2013-04-03 10:49:01.423592 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos 2013-04-03 10:49:01.423650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394147 2013-04-03 10:49:01.423666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos latest full 393798 2013-04-03 10:49:01.423686 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394148 update_from_paxos applying incremental log 394148 2013-04-03 10:49:00.443583 mon.0 10.252.24.25:6789/0 245 : [INF] pgmap v29162: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2541B/s wr, 0op/s 2013-04-03 10:49:01.423739 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 check_subs 2013-04-03 10:49:01.423776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 create_pending v 394149 2013-04-03 10:49:01.423798 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394148 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:49:01.423866 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos 2013-04-03 10:49:01.423882 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 2013-04-03 10:49:01.423900 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:01.428824 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos 2013-04-03 10:49:01.428843 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 2013-04-03 10:49:01.428861 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:01.438983 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 24 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:49:01.445331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos 2013-04-03 10:49:01.445348 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 2013-04-03 10:49:01.445363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:03.391724 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29162 check_down_pgs 2013-04-03 10:49:03.391750 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29162 v29162: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2541B/s wr, 0op/s 2013-04-03 10:49:03.392211 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:49:03.392236 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:49:03.392273 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:49:03.392304 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos 2013-04-03 10:49:03.392323 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 2013-04-03 10:49:03.392332 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394148 log 2013-04-03 10:49:03.392350 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:03.392369 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:49:03.603711 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140519 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:49:04.424437 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos 2013-04-03 10:49:04.424457 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 2013-04-03 10:49:04.424475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:04.424524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos 2013-04-03 10:49:04.424540 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 2013-04-03 10:49:04.424556 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:04.950244 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:04.950254 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a84080 from 10.252.24.88:6789/0 2013-04-03 10:49:04.950311 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 234 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:49:05.439150 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 25 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:49:07.424519 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos 2013-04-03 10:49:07.424539 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 2013-04-03 10:49:07.424557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:07.424607 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos 2013-04-03 10:49:07.424623 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 2013-04-03 10:49:07.424640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:07.603818 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140520 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:49:08.392507 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29162 v29162: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2541B/s wr, 0op/s 2013-04-03 10:49:08.392551 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:49:08.392574 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:49:08.392617 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:49:08.392648 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos 2013-04-03 10:49:08.392667 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 2013-04-03 10:49:08.392677 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394148 log 2013-04-03 10:49:08.392694 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:08.392713 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:49:08.950371 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:08.950380 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a84b80 from 10.252.24.88:6789/0 2013-04-03 10:49:08.950650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 235 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:49:09.439338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 26 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:49:09.655067 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 preprocess_query pg_stats(1 pgs tid 77 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:09.655092 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 prepare_update pg_stats(1 pgs tid 77 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:09.655097 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 prepare_pg_stats pg_stats(1 pgs tid 77 v 608) v1 from osd.1 2013-04-03 10:49:09.655103 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) 2013-04-03 10:49:09.705312 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29162 encode_pending v 29163 2013-04-03 10:49:09.752591 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29162 update_from_paxos applying incremental 29163 2013-04-03 10:49:09.752610 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 v29163: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1109B/s wr, 0op/s 2013-04-03 10:49:09.752628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 send_pg_creates to 0 pgs 2013-04-03 10:49:09.752631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 update_logger 2013-04-03 10:49:09.752643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 create_pending v 29164 2013-04-03 10:49:09.752652 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29163 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:09.752667 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 check_osd_map already seen 608 >= 608 2013-04-03 10:49:09.752669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 update_logger 2013-04-03 10:49:09.752679 7fcdffcdb700 0 log [INF] : pgmap v29163: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1109B/s wr, 0op/s 2013-04-03 10:49:09.752744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos 2013-04-03 10:49:09.752763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 2013-04-03 10:49:09.752781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:09.752824 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos 2013-04-03 10:49:09.752839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 2013-04-03 10:49:09.752847 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:09.752858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:49:09.752880 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:09.752894 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:49:09.752904 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 logging 2013-04-03 10:49:09.752680 mon.0 10.252.24.25:6789/0 246 : [INF] pgmap v29163: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1109B/s wr, 0op/s 2013-04-03 10:49:09.753377 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos 2013-04-03 10:49:09.753393 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 2013-04-03 10:49:09.753408 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:09.777557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos 2013-04-03 10:49:09.777574 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 2013-04-03 10:49:09.777588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:10.733211 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394148 encode_pending v394149 2013-04-03 10:49:10.771071 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos 2013-04-03 10:49:10.771114 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394148 2013-04-03 10:49:10.771128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos latest full 393798 2013-04-03 10:49:10.771148 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394149 update_from_paxos applying incremental log 394149 2013-04-03 10:49:09.752680 mon.0 10.252.24.25:6789/0 246 : [INF] pgmap v29163: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1109B/s wr, 0op/s 2013-04-03 10:49:10.771198 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 check_subs 2013-04-03 10:49:10.771233 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 create_pending v 394150 2013-04-03 10:49:10.771256 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394149 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:49:10.771320 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos 2013-04-03 10:49:10.771336 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 2013-04-03 10:49:10.771353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:10.775765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos 2013-04-03 10:49:10.775782 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 2013-04-03 10:49:10.775799 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:10.793460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos 2013-04-03 10:49:10.793477 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 2013-04-03 10:49:10.793492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:11.603949 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140521 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:49:12.950443 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:12.950453 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2ce1580 from 10.252.24.88:6789/0 2013-04-03 10:49:12.950508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 236 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:49:13.392836 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29163 check_down_pgs 2013-04-03 10:49:13.392870 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29163 v29163: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1109B/s wr, 0op/s 2013-04-03 10:49:13.392897 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:49:13.392923 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:49:13.392960 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:49:13.392990 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos 2013-04-03 10:49:13.393009 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 2013-04-03 10:49:13.393018 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394149 log 2013-04-03 10:49:13.393036 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:13.393054 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:49:13.439395 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 27 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:49:13.771902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos 2013-04-03 10:49:13.771922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 2013-04-03 10:49:13.771940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:13.771989 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos 2013-04-03 10:49:13.772004 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 2013-04-03 10:49:13.772019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:15.302133 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:15.302143 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34d9240 from 10.252.24.88:6789/0 2013-04-03 10:49:15.302196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 preprocess_query pg_stats(1 pgs tid 60 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:15.302220 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 prepare_update pg_stats(1 pgs tid 60 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:15.302226 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 prepare_pg_stats pg_stats(1 pgs tid 60 v 608) v1 from osd.2 2013-04-03 10:49:15.302232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:49:15.352422 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29163 encode_pending v 29164 2013-04-03 10:49:15.407558 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29163 update_from_paxos applying incremental 29164 2013-04-03 10:49:15.407573 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 v29164: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:49:15.407591 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 send_pg_creates to 0 pgs 2013-04-03 10:49:15.407593 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 update_logger 2013-04-03 10:49:15.407603 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 create_pending v 29165 2013-04-03 10:49:15.407612 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29164 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:15.407629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 check_osd_map already seen 608 >= 608 2013-04-03 10:49:15.407631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 update_logger 2013-04-03 10:49:15.407641 7fcdffcdb700 0 log [INF] : pgmap v29164: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:49:15.407699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos 2013-04-03 10:49:15.407718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 2013-04-03 10:49:15.407737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:15.407779 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos 2013-04-03 10:49:15.407794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 2013-04-03 10:49:15.407803 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:15.407813 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:49:15.407835 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:15.407845 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:49:15.407854 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 logging 2013-04-03 10:49:15.407642 mon.0 10.252.24.25:6789/0 247 : [INF] pgmap v29164: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:49:15.408512 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos 2013-04-03 10:49:15.408528 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 2013-04-03 10:49:15.408543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:15.440842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos 2013-04-03 10:49:15.440858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 2013-04-03 10:49:15.440873 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:15.604086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140522 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:49:16.396952 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394149 encode_pending v394150 2013-04-03 10:49:16.434374 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos 2013-04-03 10:49:16.434401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394149 2013-04-03 10:49:16.434418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos latest full 393798 2013-04-03 10:49:16.434435 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394150 update_from_paxos applying incremental log 394150 2013-04-03 10:49:15.407642 mon.0 10.252.24.25:6789/0 247 : [INF] pgmap v29164: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:49:16.434479 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 check_subs 2013-04-03 10:49:16.434516 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 create_pending v 394151 2013-04-03 10:49:16.434538 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394150 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:49:16.434603 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos 2013-04-03 10:49:16.434618 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 2013-04-03 10:49:16.434637 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:16.437042 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos 2013-04-03 10:49:16.437058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 2013-04-03 10:49:16.437072 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:16.447049 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos 2013-04-03 10:49:16.447066 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 2013-04-03 10:49:16.447083 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:16.950571 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:16.950581 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2ce1b00 from 10.252.24.88:6789/0 2013-04-03 10:49:16.950637 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 237 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:49:17.439555 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 28 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:49:18.393197 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29164 check_down_pgs 2013-04-03 10:49:18.393228 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29164 v29164: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:49:18.393256 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:49:18.393280 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:49:18.393326 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:49:18.393359 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos 2013-04-03 10:49:18.393378 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 2013-04-03 10:49:18.393387 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394150 log 2013-04-03 10:49:18.393405 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:18.393423 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:49:19.435127 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos 2013-04-03 10:49:19.435147 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 2013-04-03 10:49:19.435165 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:19.435520 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos 2013-04-03 10:49:19.435540 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 2013-04-03 10:49:19.435555 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:19.604139 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140523 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:49:20.950805 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:20.950814 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a528c0 from 10.252.24.88:6789/0 2013-04-03 10:49:20.950870 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 238 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:49:21.439685 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 29 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:49:22.435236 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos 2013-04-03 10:49:22.435256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 2013-04-03 10:49:22.435274 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:22.435322 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos 2013-04-03 10:49:22.435338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 2013-04-03 10:49:22.435355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:23.393560 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29164 v29164: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s 2013-04-03 10:49:23.393605 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:49:23.393628 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:49:23.393671 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:49:23.393702 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos 2013-04-03 10:49:23.393721 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 2013-04-03 10:49:23.393729 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394150 log 2013-04-03 10:49:23.393745 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:23.393762 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:49:23.604264 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140524 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:49:23.857867 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 preprocess_query pg_stats(1 pgs tid 5 v 608) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:49:23.857892 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 prepare_update pg_stats(1 pgs tid 5 v 608) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:49:23.857897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 prepare_pg_stats pg_stats(1 pgs tid 5 v 608) v1 from osd.3 2013-04-03 10:49:23.857903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:49:23.908095 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29164 encode_pending v 29165 2013-04-03 10:49:23.952060 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29164 update_from_paxos applying incremental 29165 2013-04-03 10:49:23.952076 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 v29165: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1225B/s wr, 0op/s 2013-04-03 10:49:23.952094 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 send_pg_creates to 0 pgs 2013-04-03 10:49:23.952096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 update_logger 2013-04-03 10:49:23.952106 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 create_pending v 29166 2013-04-03 10:49:23.952114 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29165 _updated_stats for osd.3 10.252.24.88:6804/18376 2013-04-03 10:49:23.952133 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 check_osd_map already seen 608 >= 608 2013-04-03 10:49:23.952136 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 update_logger 2013-04-03 10:49:23.952147 7fcdffcdb700 0 log [INF] : pgmap v29165: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1225B/s wr, 0op/s 2013-04-03 10:49:23.952206 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos 2013-04-03 10:49:23.952225 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 2013-04-03 10:49:23.952245 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:23.952289 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos 2013-04-03 10:49:23.952304 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 2013-04-03 10:49:23.952313 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:23.952323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:49:23.952629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:23.952645 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:49:23.952655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 logging 2013-04-03 10:49:23.952148 mon.0 10.252.24.25:6789/0 248 : [INF] pgmap v29165: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1225B/s wr, 0op/s 2013-04-03 10:49:23.962161 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos 2013-04-03 10:49:23.962177 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 2013-04-03 10:49:23.962192 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:23.977043 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos 2013-04-03 10:49:23.977059 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 2013-04-03 10:49:23.977074 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:24.656740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 preprocess_query pg_stats(1 pgs tid 78 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:24.656763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 prepare_update pg_stats(1 pgs tid 78 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:24.656768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 prepare_pg_stats pg_stats(1 pgs tid 78 v 608) v1 from osd.1 2013-04-03 10:49:24.656773 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) 2013-04-03 10:49:24.933867 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394150 encode_pending v394151 2013-04-03 10:49:24.935260 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29165 encode_pending v 29166 2013-04-03 10:49:25.060850 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos 2013-04-03 10:49:25.060881 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos version 394151 summary v 394150 2013-04-03 10:49:25.060895 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos latest full 393798 2013-04-03 10:49:25.060913 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394151 update_from_paxos applying incremental log 394151 2013-04-03 10:49:23.952148 mon.0 10.252.24.25:6789/0 248 : [INF] pgmap v29165: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1225B/s wr, 0op/s 2013-04-03 10:49:25.060956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 check_subs 2013-04-03 10:49:25.060992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 create_pending v 394152 2013-04-03 10:49:25.061014 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394151 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:49:25.110395 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29165 update_from_paxos applying incremental 29166 2013-04-03 10:49:25.110413 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 v29166: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s 2013-04-03 10:49:25.110432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 send_pg_creates to 0 pgs 2013-04-03 10:49:25.110435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 update_logger 2013-04-03 10:49:25.110444 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 create_pending v 29167 2013-04-03 10:49:25.110453 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29166 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:25.110468 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 check_osd_map already seen 608 >= 608 2013-04-03 10:49:25.110470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 update_logger 2013-04-03 10:49:25.110480 7fcdffcdb700 0 log [INF] : pgmap v29166: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s 2013-04-03 10:49:25.110537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos 2013-04-03 10:49:25.110554 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos version 394151 summary v 394151 2013-04-03 10:49:25.110571 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:25.110617 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos 2013-04-03 10:49:25.110632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos version 394151 summary v 394151 2013-04-03 10:49:25.110640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:25.110651 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:49:25.110669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:25.110680 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:49:25.110689 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 logging 2013-04-03 10:49:25.110481 mon.0 10.252.24.25:6789/0 249 : [INF] pgmap v29166: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s 2013-04-03 10:49:25.111617 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos 2013-04-03 10:49:25.111633 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos version 394151 summary v 394151 2013-04-03 10:49:25.111648 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:25.122086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos 2013-04-03 10:49:25.122102 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos version 394151 summary v 394151 2013-04-03 10:49:25.122117 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:25.122303 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:25.122312 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a51000 from 10.252.24.88:6789/0 2013-04-03 10:49:25.122344 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 239 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:49:25.439743 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 30 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:49:26.086768 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394151 encode_pending v394152 2013-04-03 10:49:26.124055 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos 2013-04-03 10:49:26.124082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394151 2013-04-03 10:49:26.124096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos latest full 393798 2013-04-03 10:49:26.124113 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394152 update_from_paxos applying incremental log 394152 2013-04-03 10:49:25.110481 mon.0 10.252.24.25:6789/0 249 : [INF] pgmap v29166: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s 2013-04-03 10:49:26.124156 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 check_subs 2013-04-03 10:49:26.124192 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 create_pending v 394153 2013-04-03 10:49:26.124217 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394152 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:49:26.124287 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos 2013-04-03 10:49:26.124303 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 2013-04-03 10:49:26.124319 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:26.132609 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos 2013-04-03 10:49:26.132639 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 2013-04-03 10:49:26.132655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:26.133434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos 2013-04-03 10:49:26.133450 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 2013-04-03 10:49:26.133467 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:27.604432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140525 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:49:28.393901 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29166 check_down_pgs 2013-04-03 10:49:28.393925 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29166 v29166: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s 2013-04-03 10:49:28.393953 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:49:28.393977 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:49:28.394013 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:49:28.394044 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos 2013-04-03 10:49:28.394063 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 2013-04-03 10:49:28.394072 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394152 log 2013-04-03 10:49:28.394089 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:28.394111 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:49:28.951178 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:28.951192 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52600 from 10.252.24.88:6789/0 2013-04-03 10:49:28.951246 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 240 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:49:29.124794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos 2013-04-03 10:49:29.124815 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 2013-04-03 10:49:29.124833 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:29.124886 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos 2013-04-03 10:49:29.124902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 2013-04-03 10:49:29.125200 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:29.439844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 31 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:49:29.657227 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 preprocess_query pg_stats(1 pgs tid 79 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:29.657250 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 prepare_update pg_stats(1 pgs tid 79 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:29.657255 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 prepare_pg_stats pg_stats(1 pgs tid 79 v 608) v1 from osd.1 2013-04-03 10:49:29.657260 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) 2013-04-03 10:49:29.707467 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29166 encode_pending v 29167 2013-04-03 10:49:29.748292 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29166 update_from_paxos applying incremental 29167 2013-04-03 10:49:29.748313 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 v29167: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5650B/s wr, 0op/s 2013-04-03 10:49:29.748331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 send_pg_creates to 0 pgs 2013-04-03 10:49:29.748334 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 update_logger 2013-04-03 10:49:29.748344 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 create_pending v 29168 2013-04-03 10:49:29.748353 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29167 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:29.748369 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 check_osd_map already seen 608 >= 608 2013-04-03 10:49:29.748371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 update_logger 2013-04-03 10:49:29.748382 7fcdffcdb700 0 log [INF] : pgmap v29167: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5650B/s wr, 0op/s 2013-04-03 10:49:29.748447 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos 2013-04-03 10:49:29.748467 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 2013-04-03 10:49:29.748485 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:29.748531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos 2013-04-03 10:49:29.748546 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 2013-04-03 10:49:29.748554 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:29.748564 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:49:29.748588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:29.748600 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:49:29.748610 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 logging 2013-04-03 10:49:29.748383 mon.0 10.252.24.25:6789/0 250 : [INF] pgmap v29167: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5650B/s wr, 0op/s 2013-04-03 10:49:29.759748 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos 2013-04-03 10:49:29.759768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 2013-04-03 10:49:29.759785 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:29.761664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos 2013-04-03 10:49:29.761680 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 2013-04-03 10:49:29.761699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:29.879420 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 preprocess_query pg_stats(1 pgs tid 414 v 608) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:49:29.879443 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 prepare_update pg_stats(1 pgs tid 414 v 608) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:49:29.879449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 prepare_pg_stats pg_stats(1 pgs tid 414 v 608) v1 from osd.0 2013-04-03 10:49:29.879456 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 got osd.0 osd_stat(1143 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:49:30.303619 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:30.303628 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34d9480 from 10.252.24.88:6789/0 2013-04-03 10:49:30.303660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 preprocess_query pg_stats(1 pgs tid 61 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:30.303681 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 prepare_update pg_stats(1 pgs tid 61 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:30.303686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 prepare_pg_stats pg_stats(1 pgs tid 61 v 608) v1 from osd.2 2013-04-03 10:49:30.303690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:49:30.734655 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29167 encode_pending v 29168 2013-04-03 10:49:30.735915 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394152 encode_pending v394153 2013-04-03 10:49:30.766829 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29167 update_from_paxos applying incremental 29168 2013-04-03 10:49:30.766853 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 v29168: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 7415B/s wr, 1op/s 2013-04-03 10:49:30.766872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 send_pg_creates to 0 pgs 2013-04-03 10:49:30.766874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 update_logger 2013-04-03 10:49:30.766885 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 create_pending v 29169 2013-04-03 10:49:30.766894 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29168 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:49:30.766909 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29168 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:30.766922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 check_osd_map already seen 608 >= 608 2013-04-03 10:49:30.766924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 update_logger 2013-04-03 10:49:30.766934 7fcdffcdb700 0 log [INF] : pgmap v29168: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 7415B/s wr, 1op/s 2013-04-03 10:49:30.798109 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos 2013-04-03 10:49:30.798133 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394152 2013-04-03 10:49:30.798147 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos latest full 393798 2013-04-03 10:49:30.798164 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394153 update_from_paxos applying incremental log 394153 2013-04-03 10:49:29.748383 mon.0 10.252.24.25:6789/0 250 : [INF] pgmap v29167: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5650B/s wr, 0op/s 2013-04-03 10:49:30.798208 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 check_subs 2013-04-03 10:49:30.798246 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 create_pending v 394154 2013-04-03 10:49:30.798270 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394153 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:49:30.798301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos 2013-04-03 10:49:30.798316 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 2013-04-03 10:49:30.798326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:30.798338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:49:30.798355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 nothing new 2013-04-03 10:49:30.798402 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos 2013-04-03 10:49:30.798418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 2013-04-03 10:49:30.798435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:30.811525 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos 2013-04-03 10:49:30.811541 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 2013-04-03 10:49:30.811556 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:30.816526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos 2013-04-03 10:49:30.816543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 2013-04-03 10:49:30.816558 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:31.604565 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140526 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:49:32.951185 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:32.951195 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2146b80 from 10.252.24.88:6789/0 2013-04-03 10:49:32.951243 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 241 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:49:33.394237 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29168 check_down_pgs 2013-04-03 10:49:33.394269 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29168 v29168: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 7415B/s wr, 1op/s 2013-04-03 10:49:33.394293 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:49:33.394318 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:49:33.394355 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:49:33.394386 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos 2013-04-03 10:49:33.394405 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 2013-04-03 10:49:33.394414 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394153 log 2013-04-03 10:49:33.394431 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:33.395087 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:49:33.439940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 32 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:49:33.798850 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos 2013-04-03 10:49:33.798871 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 2013-04-03 10:49:33.798889 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:33.799029 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos 2013-04-03 10:49:33.799048 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 2013-04-03 10:49:33.799064 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:34.572681 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:34.572693 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34d9480 from 10.252.24.88:6789/0 2013-04-03 10:49:34.572745 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 preprocess_query pg_stats(0 pgs tid 65 v 608) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:49:34.572769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 prepare_update pg_stats(0 pgs tid 65 v 608) v1 from osd.4 10.252.24.32:6800/13473 2013-04-03 10:49:34.572775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 prepare_pg_stats pg_stats(0 pgs tid 65 v 608) v1 from osd.4 2013-04-03 10:49:34.572781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 got osd.4 osd_stat(1049 MB used, 56607 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[])) 2013-04-03 10:49:34.622964 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29168 encode_pending v 29169 2013-04-03 10:49:34.858007 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29168 update_from_paxos applying incremental 29169 2013-04-03 10:49:34.858018 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 v29169: 240 pgs: 240 active+clean; 256 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 2916B/s wr, 0op/s 2013-04-03 10:49:34.858035 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 send_pg_creates to 0 pgs 2013-04-03 10:49:34.858038 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 update_logger 2013-04-03 10:49:34.858047 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 create_pending v 29170 2013-04-03 10:49:34.858056 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29169 _updated_stats for osd.4 10.252.24.32:6800/13473 2013-04-03 10:49:34.858093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 preprocess_query pg_stats(2 pgs tid 80 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:34.858118 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 prepare_update pg_stats(2 pgs tid 80 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:34.858123 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 prepare_pg_stats pg_stats(2 pgs tid 80 v 608) v1 from osd.1 2013-04-03 10:49:34.858129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) 2013-04-03 10:49:34.858152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 check_osd_map already seen 608 >= 608 2013-04-03 10:49:34.858156 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 update_logger 2013-04-03 10:49:34.858167 7fcdffcdb700 0 log [INF] : pgmap v29169: 240 pgs: 240 active+clean; 256 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 2916B/s wr, 0op/s 2013-04-03 10:49:34.861844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos 2013-04-03 10:49:34.861861 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 2013-04-03 10:49:34.861871 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:34.861881 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:49:34.861923 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:34.861938 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:49:34.861948 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 logging 2013-04-03 10:49:30.766937 mon.0 10.252.24.25:6789/0 251 : [INF] pgmap v29168: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 7415B/s wr, 1op/s 2013-04-03 10:49:34.861969 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 logging 2013-04-03 10:49:34.858168 mon.0 10.252.24.25:6789/0 252 : [INF] pgmap v29169: 240 pgs: 240 active+clean; 256 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 2916B/s wr, 0op/s 2013-04-03 10:49:34.879935 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 preprocess_query pg_stats(1 pgs tid 415 v 608) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:49:34.879956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 prepare_update pg_stats(1 pgs tid 415 v 608) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:49:34.879960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 prepare_pg_stats pg_stats(1 pgs tid 415 v 608) v1 from osd.0 2013-04-03 10:49:34.879964 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 got osd.0 osd_stat(1143 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1143 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:49:34.964328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos 2013-04-03 10:49:34.964346 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 2013-04-03 10:49:34.964364 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:34.964990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos 2013-04-03 10:49:34.965006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 2013-04-03 10:49:34.965023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:35.033216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos 2013-04-03 10:49:35.033234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 2013-04-03 10:49:35.033251 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:35.304151 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:35.304159 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x3598d80 from 10.252.24.88:6789/0 2013-04-03 10:49:35.304207 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 preprocess_query pg_stats(1 pgs tid 62 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:35.304231 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 prepare_update pg_stats(1 pgs tid 62 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:35.304237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 prepare_pg_stats pg_stats(1 pgs tid 62 v 608) v1 from osd.2 2013-04-03 10:49:35.304243 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:49:35.604661 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140527 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:49:35.650860 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394153 encode_pending v394154 2013-04-03 10:49:35.652420 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29169 encode_pending v 29170 2013-04-03 10:49:35.693513 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos 2013-04-03 10:49:35.693556 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos version 394154 summary v 394153 2013-04-03 10:49:35.693570 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos latest full 393798 2013-04-03 10:49:35.693590 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394154 update_from_paxos applying incremental log 394154 2013-04-03 10:49:30.766937 mon.0 10.252.24.25:6789/0 251 : [INF] pgmap v29168: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 7415B/s wr, 1op/s 2013-04-03 10:49:35.693617 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394154 update_from_paxos applying incremental log 394154 2013-04-03 10:49:34.858168 mon.0 10.252.24.25:6789/0 252 : [INF] pgmap v29169: 240 pgs: 240 active+clean; 256 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 2916B/s wr, 0op/s 2013-04-03 10:49:35.693655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 check_subs 2013-04-03 10:49:35.693691 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 create_pending v 394155 2013-04-03 10:49:35.694306 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394154 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:49:35.735188 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29169 update_from_paxos applying incremental 29170 2013-04-03 10:49:35.735218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 v29170: 240 pgs: 240 active+clean; 256 MB data, 5664 MB used, 1067 GB / 1079 GB avail; 74960B/s wr, 1op/s 2013-04-03 10:49:35.735239 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 send_pg_creates to 0 pgs 2013-04-03 10:49:35.735242 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 update_logger 2013-04-03 10:49:35.735253 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 create_pending v 29171 2013-04-03 10:49:35.735262 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29170 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:35.735278 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29170 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:49:35.735293 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29170 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:35.735306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 check_osd_map already seen 608 >= 608 2013-04-03 10:49:35.735309 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 update_logger 2013-04-03 10:49:35.735319 7fcdffcdb700 0 log [INF] : pgmap v29170: 240 pgs: 240 active+clean; 256 MB data, 5664 MB used, 1067 GB / 1079 GB avail; 74960B/s wr, 1op/s 2013-04-03 10:49:35.735379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos 2013-04-03 10:49:35.735398 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos version 394154 summary v 394154 2013-04-03 10:49:35.735417 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:35.735455 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos 2013-04-03 10:49:35.735471 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos version 394154 summary v 394154 2013-04-03 10:49:35.735480 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:35.735492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:49:35.735513 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:35.735525 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:49:35.735535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 logging 2013-04-03 10:49:35.735320 mon.0 10.252.24.25:6789/0 253 : [INF] pgmap v29170: 240 pgs: 240 active+clean; 256 MB data, 5664 MB used, 1067 GB / 1079 GB avail; 74960B/s wr, 1op/s 2013-04-03 10:49:35.748328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos 2013-04-03 10:49:35.748346 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos version 394154 summary v 394154 2013-04-03 10:49:35.748361 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:35.760218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos 2013-04-03 10:49:35.760236 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos version 394154 summary v 394154 2013-04-03 10:49:35.760252 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:36.723069 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394154 encode_pending v394155 2013-04-03 10:49:36.764228 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos 2013-04-03 10:49:36.764264 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394154 2013-04-03 10:49:36.764278 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos latest full 393798 2013-04-03 10:49:36.764296 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394155 update_from_paxos applying incremental log 394155 2013-04-03 10:49:35.735320 mon.0 10.252.24.25:6789/0 253 : [INF] pgmap v29170: 240 pgs: 240 active+clean; 256 MB data, 5664 MB used, 1067 GB / 1079 GB avail; 74960B/s wr, 1op/s 2013-04-03 10:49:36.764339 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 check_subs 2013-04-03 10:49:36.764379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 create_pending v 394156 2013-04-03 10:49:36.764403 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394155 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:49:36.764479 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos 2013-04-03 10:49:36.764498 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 2013-04-03 10:49:36.764517 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:36.772155 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos 2013-04-03 10:49:36.772173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 2013-04-03 10:49:36.772188 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:36.776261 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos 2013-04-03 10:49:36.776280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 2013-04-03 10:49:36.776296 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:36.951246 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:36.951255 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2ce2340 from 10.252.24.88:6789/0 2013-04-03 10:49:36.951311 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 242 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:49:37.440137 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 33 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:49:38.395223 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29170 check_down_pgs 2013-04-03 10:49:38.395256 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29170 v29170: 240 pgs: 240 active+clean; 256 MB data, 5664 MB used, 1067 GB / 1079 GB avail; 74960B/s wr, 1op/s 2013-04-03 10:49:38.395284 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:49:38.395308 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:49:38.395344 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:49:38.395375 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos 2013-04-03 10:49:38.395394 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 2013-04-03 10:49:38.395403 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394155 log 2013-04-03 10:49:38.395421 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:38.395439 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:49:38.859806 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 preprocess_query pg_stats(2 pgs tid 6 v 608) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:49:38.859836 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 prepare_update pg_stats(2 pgs tid 6 v 608) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:49:38.859841 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 prepare_pg_stats pg_stats(2 pgs tid 6 v 608) v1 from osd.3 2013-04-03 10:49:38.859847 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:49:38.910034 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29170 encode_pending v 29171 2013-04-03 10:49:39.063512 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29170 update_from_paxos applying incremental 29171 2013-04-03 10:49:39.063530 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 v29171: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 86227B/s wr, 1op/s 2013-04-03 10:49:39.063548 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 send_pg_creates to 0 pgs 2013-04-03 10:49:39.063552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 update_logger 2013-04-03 10:49:39.063563 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 create_pending v 29172 2013-04-03 10:49:39.063571 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29171 _updated_stats for osd.3 10.252.24.88:6804/18376 2013-04-03 10:49:39.063587 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 check_osd_map already seen 608 >= 608 2013-04-03 10:49:39.063589 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 update_logger 2013-04-03 10:49:39.063599 7fcdffcdb700 0 log [INF] : pgmap v29171: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 86227B/s wr, 1op/s 2013-04-03 10:49:39.063657 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos 2013-04-03 10:49:39.063677 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 2013-04-03 10:49:39.063694 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:39.063754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos 2013-04-03 10:49:39.063769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 2013-04-03 10:49:39.063777 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:39.063789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:49:39.063811 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:39.063822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:49:39.063831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 logging 2013-04-03 10:49:39.063600 mon.0 10.252.24.25:6789/0 254 : [INF] pgmap v29171: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 86227B/s wr, 1op/s 2013-04-03 10:49:39.064250 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos 2013-04-03 10:49:39.064266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 2013-04-03 10:49:39.064280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:39.105260 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos 2013-04-03 10:49:39.105277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 2013-04-03 10:49:39.105292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:39.604778 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140528 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:49:39.658397 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 preprocess_query pg_stats(3 pgs tid 81 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:39.658418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 prepare_update pg_stats(3 pgs tid 81 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:39.658422 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 prepare_pg_stats pg_stats(3 pgs tid 81 v 608) v1 from osd.1 2013-04-03 10:49:39.658427 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 got osd.1 osd_stat(1104 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) 2013-04-03 10:49:39.880543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 preprocess_query pg_stats(1 pgs tid 416 v 608) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:49:39.880568 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 prepare_update pg_stats(1 pgs tid 416 v 608) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:49:39.880573 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 prepare_pg_stats pg_stats(1 pgs tid 416 v 608) v1 from osd.0 2013-04-03 10:49:39.880579 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 got osd.0 osd_stat(1146 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1143 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:49:39.938912 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394155 encode_pending v394156 2013-04-03 10:49:39.940136 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29171 encode_pending v 29172 2013-04-03 10:49:39.967726 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos 2013-04-03 10:49:39.967750 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos version 394156 summary v 394155 2013-04-03 10:49:39.967764 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos latest full 393798 2013-04-03 10:49:39.967782 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394156 update_from_paxos applying incremental log 394156 2013-04-03 10:49:39.063600 mon.0 10.252.24.25:6789/0 254 : [INF] pgmap v29171: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 86227B/s wr, 1op/s 2013-04-03 10:49:39.967826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 check_subs 2013-04-03 10:49:39.967861 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 create_pending v 394157 2013-04-03 10:49:39.967883 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394156 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:49:40.005074 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29171 update_from_paxos applying incremental 29172 2013-04-03 10:49:40.005096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 v29172: 240 pgs: 240 active+clean; 253 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 21016B/s wr, 3op/s 2013-04-03 10:49:40.005117 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 send_pg_creates to 0 pgs 2013-04-03 10:49:40.005119 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 update_logger 2013-04-03 10:49:40.005129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 create_pending v 29173 2013-04-03 10:49:40.005138 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29172 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:40.005161 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29172 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:49:40.005173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 check_osd_map already seen 608 >= 608 2013-04-03 10:49:40.005175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 update_logger 2013-04-03 10:49:40.005185 7fcdffcdb700 0 log [INF] : pgmap v29172: 240 pgs: 240 active+clean; 253 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 21016B/s wr, 3op/s 2013-04-03 10:49:40.005284 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos 2013-04-03 10:49:40.005302 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos version 394156 summary v 394156 2013-04-03 10:49:40.005319 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:40.005363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos 2013-04-03 10:49:40.005378 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos version 394156 summary v 394156 2013-04-03 10:49:40.005386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:40.005396 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:49:40.005413 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:40.005425 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:49:40.005434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 logging 2013-04-03 10:49:40.005186 mon.0 10.252.24.25:6789/0 255 : [INF] pgmap v29172: 240 pgs: 240 active+clean; 253 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 21016B/s wr, 3op/s 2013-04-03 10:49:40.009234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos 2013-04-03 10:49:40.009250 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos version 394156 summary v 394156 2013-04-03 10:49:40.009265 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:40.030505 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos 2013-04-03 10:49:40.030522 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos version 394156 summary v 394156 2013-04-03 10:49:40.030537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:40.304921 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:40.304930 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x3598d80 from 10.252.24.88:6789/0 2013-04-03 10:49:40.304980 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 preprocess_query pg_stats(2 pgs tid 63 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:40.305003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 prepare_update pg_stats(2 pgs tid 63 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:40.305009 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 prepare_pg_stats pg_stats(2 pgs tid 63 v 608) v1 from osd.2 2013-04-03 10:49:40.305015 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 got osd.2 osd_stat(1182 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:49:40.951405 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:40.951415 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83dc0 from 10.252.24.88:6789/0 2013-04-03 10:49:40.951470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 243 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:49:40.999688 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29172 encode_pending v 29173 2013-04-03 10:49:41.001084 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394156 encode_pending v394157 2013-04-03 10:49:41.034590 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29172 update_from_paxos applying incremental 29173 2013-04-03 10:49:41.034611 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 v29173: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1698KB/s wr, 8op/s 2013-04-03 10:49:41.034631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 send_pg_creates to 0 pgs 2013-04-03 10:49:41.034633 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 update_logger 2013-04-03 10:49:41.034645 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 create_pending v 29174 2013-04-03 10:49:41.034654 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29173 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:41.034671 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 check_osd_map already seen 608 >= 608 2013-04-03 10:49:41.034675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 update_logger 2013-04-03 10:49:41.034685 7fcdffcdb700 0 log [INF] : pgmap v29173: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1698KB/s wr, 8op/s 2013-04-03 10:49:41.068957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos 2013-04-03 10:49:41.068981 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394156 2013-04-03 10:49:41.068995 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos latest full 393798 2013-04-03 10:49:41.069012 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394157 update_from_paxos applying incremental log 394157 2013-04-03 10:49:40.005186 mon.0 10.252.24.25:6789/0 255 : [INF] pgmap v29172: 240 pgs: 240 active+clean; 253 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 21016B/s wr, 3op/s 2013-04-03 10:49:41.069058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 check_subs 2013-04-03 10:49:41.069093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 create_pending v 394158 2013-04-03 10:49:41.069116 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394157 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:49:41.069149 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos 2013-04-03 10:49:41.069164 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 2013-04-03 10:49:41.069173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:41.069184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:49:41.069200 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 nothing new 2013-04-03 10:49:41.069246 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos 2013-04-03 10:49:41.069262 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 2013-04-03 10:49:41.069281 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:41.080458 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos 2013-04-03 10:49:41.080474 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 2013-04-03 10:49:41.080489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:41.101136 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos 2013-04-03 10:49:41.101153 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 2013-04-03 10:49:41.101168 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:41.440185 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 34 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:49:41.561208 7fcdfdfcc700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4780 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33219/0) 2013-04-03 10:49:43.395574 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29173 check_down_pgs 2013-04-03 10:49:43.395605 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29173 v29173: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1698KB/s wr, 8op/s 2013-04-03 10:49:43.395633 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:49:43.395655 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:49:43.395692 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:49:43.395723 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos 2013-04-03 10:49:43.395741 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 2013-04-03 10:49:43.395750 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394157 log 2013-04-03 10:49:43.395768 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:43.395786 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:49:43.604908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140529 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:49:44.069803 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos 2013-04-03 10:49:44.069825 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 2013-04-03 10:49:44.069842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:44.069891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos 2013-04-03 10:49:44.069907 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 2013-04-03 10:49:44.069923 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:44.659019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 preprocess_query pg_stats(1 pgs tid 82 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:44.659042 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 prepare_update pg_stats(1 pgs tid 82 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:44.659046 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 prepare_pg_stats pg_stats(1 pgs tid 82 v 608) v1 from osd.1 2013-04-03 10:49:44.659052 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 got osd.1 osd_stat(1104 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1104 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) 2013-04-03 10:49:44.709255 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29173 encode_pending v 29174 2013-04-03 10:49:44.752253 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29173 update_from_paxos applying incremental 29174 2013-04-03 10:49:44.752272 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 v29174: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 729KB/s wr, 1op/s 2013-04-03 10:49:44.752290 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 send_pg_creates to 0 pgs 2013-04-03 10:49:44.752295 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 update_logger 2013-04-03 10:49:44.752306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 create_pending v 29175 2013-04-03 10:49:44.752315 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29174 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:44.752331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 check_osd_map already seen 608 >= 608 2013-04-03 10:49:44.752333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 update_logger 2013-04-03 10:49:44.752343 7fcdffcdb700 0 log [INF] : pgmap v29174: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 729KB/s wr, 1op/s 2013-04-03 10:49:44.752408 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos 2013-04-03 10:49:44.752428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 2013-04-03 10:49:44.752448 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:44.752492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos 2013-04-03 10:49:44.752507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 2013-04-03 10:49:44.752516 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:44.752527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:49:44.752555 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:44.752567 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:49:44.752576 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 logging 2013-04-03 10:49:41.034687 mon.0 10.252.24.25:6789/0 256 : [INF] pgmap v29173: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1698KB/s wr, 8op/s 2013-04-03 10:49:44.752596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 logging 2013-04-03 10:49:44.752344 mon.0 10.252.24.25:6789/0 257 : [INF] pgmap v29174: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 729KB/s wr, 1op/s 2013-04-03 10:49:44.761478 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos 2013-04-03 10:49:44.761494 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 2013-04-03 10:49:44.761509 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:44.770064 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos 2013-04-03 10:49:44.770080 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 2013-04-03 10:49:44.770095 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:44.951543 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:44.951553 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a848c0 from 10.252.24.88:6789/0 2013-04-03 10:49:44.951606 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 244 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:49:45.305057 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:45.305063 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x3598240 from 10.252.24.88:6789/0 2013-04-03 10:49:45.305097 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 preprocess_query pg_stats(1 pgs tid 64 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:45.305119 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 prepare_update pg_stats(1 pgs tid 64 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:45.305125 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 prepare_pg_stats pg_stats(1 pgs tid 64 v 608) v1 from osd.2 2013-04-03 10:49:45.305131 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 got osd.2 osd_stat(1182 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1182 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:49:45.440299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 35 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:49:45.744667 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29174 encode_pending v 29175 2013-04-03 10:49:45.746120 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394157 encode_pending v394158 2013-04-03 10:49:45.787321 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29174 update_from_paxos applying incremental 29175 2013-04-03 10:49:45.787340 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 v29175: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9063B/s wr, 1op/s 2013-04-03 10:49:45.787360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 send_pg_creates to 0 pgs 2013-04-03 10:49:45.787362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 update_logger 2013-04-03 10:49:45.787372 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 create_pending v 29176 2013-04-03 10:49:45.787381 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29175 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:49:45.787401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 check_osd_map already seen 608 >= 608 2013-04-03 10:49:45.787403 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 update_logger 2013-04-03 10:49:45.787413 7fcdffcdb700 0 log [INF] : pgmap v29175: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9063B/s wr, 1op/s 2013-04-03 10:49:45.821180 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos 2013-04-03 10:49:45.821203 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394157 2013-04-03 10:49:45.821218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos latest full 393798 2013-04-03 10:49:45.821234 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394158 update_from_paxos applying incremental log 394158 2013-04-03 10:49:41.034687 mon.0 10.252.24.25:6789/0 256 : [INF] pgmap v29173: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1698KB/s wr, 8op/s 2013-04-03 10:49:45.821260 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394158 update_from_paxos applying incremental log 394158 2013-04-03 10:49:44.752344 mon.0 10.252.24.25:6789/0 257 : [INF] pgmap v29174: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 729KB/s wr, 1op/s 2013-04-03 10:49:45.821296 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 check_subs 2013-04-03 10:49:45.821332 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 create_pending v 394159 2013-04-03 10:49:45.821354 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394158 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:49:45.821386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos 2013-04-03 10:49:45.821401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 2013-04-03 10:49:45.821412 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:45.821422 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:49:45.821439 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 nothing new 2013-04-03 10:49:45.821487 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos 2013-04-03 10:49:45.821502 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 2013-04-03 10:49:45.821519 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:45.829474 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos 2013-04-03 10:49:45.829490 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 2013-04-03 10:49:45.829504 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:45.838322 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos 2013-04-03 10:49:45.838338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 2013-04-03 10:49:45.838353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:47.605108 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140530 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:49:48.395939 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29175 check_down_pgs 2013-04-03 10:49:48.395964 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29175 v29175: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9063B/s wr, 1op/s 2013-04-03 10:49:48.395991 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:49:48.396014 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:49:48.396050 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:49:48.396081 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos 2013-04-03 10:49:48.396099 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 2013-04-03 10:49:48.396108 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394158 log 2013-04-03 10:49:48.396126 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:48.396145 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:49:48.822046 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos 2013-04-03 10:49:48.822065 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 2013-04-03 10:49:48.822083 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:48.822132 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos 2013-04-03 10:49:48.822147 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 2013-04-03 10:49:48.822162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:48.951584 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:48.951592 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34ad8c0 from 10.252.24.88:6789/0 2013-04-03 10:49:48.951625 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 245 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:49:49.440458 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 36 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:49:51.577228 7fcdfdfcc700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3250500 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33220/0) 2013-04-03 10:49:51.605196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140531 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:49:51.822061 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos 2013-04-03 10:49:51.822083 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 2013-04-03 10:49:51.822101 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:51.822156 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos 2013-04-03 10:49:51.822172 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 2013-04-03 10:49:51.822187 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:52.951721 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:52.951730 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34ac580 from 10.252.24.88:6789/0 2013-04-03 10:49:52.951779 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 246 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:49:53.396277 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29175 v29175: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9063B/s wr, 1op/s 2013-04-03 10:49:53.396319 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:49:53.396342 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:49:53.396384 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:49:53.396417 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos 2013-04-03 10:49:53.396436 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 2013-04-03 10:49:53.396445 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394158 log 2013-04-03 10:49:53.396463 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:53.396482 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:49:53.440534 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 37 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:49:54.660220 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 preprocess_query pg_stats(1 pgs tid 83 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:54.660248 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 prepare_update pg_stats(1 pgs tid 83 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:54.660253 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 prepare_pg_stats pg_stats(1 pgs tid 83 v 608) v1 from osd.1 2013-04-03 10:49:54.660259 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 got osd.1 osd_stat(1104 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1104 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) 2013-04-03 10:49:54.710433 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29175 encode_pending v 29176 2013-04-03 10:49:54.763725 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29175 update_from_paxos applying incremental 29176 2013-04-03 10:49:54.763744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 v29176: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2969B/s wr, 0op/s 2013-04-03 10:49:54.763763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 send_pg_creates to 0 pgs 2013-04-03 10:49:54.763765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 update_logger 2013-04-03 10:49:54.763775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 create_pending v 29177 2013-04-03 10:49:54.763786 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29176 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:49:54.763801 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 check_osd_map already seen 608 >= 608 2013-04-03 10:49:54.763803 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 update_logger 2013-04-03 10:49:54.763813 7fcdffcdb700 0 log [INF] : pgmap v29176: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2969B/s wr, 0op/s 2013-04-03 10:49:54.763879 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos 2013-04-03 10:49:54.763899 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 2013-04-03 10:49:54.763917 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:54.763960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos 2013-04-03 10:49:54.763975 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 2013-04-03 10:49:54.763984 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:54.763994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 preprocess_log log(2 entries) v1 from mon.0 2013-04-03 10:49:54.764023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:49:54.764037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 prepare_log log(2 entries) v1 from mon.0 2013-04-03 10:49:54.764047 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 logging 2013-04-03 10:49:45.787415 mon.0 10.252.24.25:6789/0 258 : [INF] pgmap v29175: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9063B/s wr, 1op/s 2013-04-03 10:49:54.764066 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 logging 2013-04-03 10:49:54.763814 mon.0 10.252.24.25:6789/0 259 : [INF] pgmap v29176: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2969B/s wr, 0op/s 2013-04-03 10:49:54.769461 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos 2013-04-03 10:49:54.769477 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 2013-04-03 10:49:54.769492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:54.788690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos 2013-04-03 10:49:54.788706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 2013-04-03 10:49:54.788720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:55.605341 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140532 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:49:55.757449 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394158 encode_pending v394159 2013-04-03 10:49:55.790484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos 2013-04-03 10:49:55.790511 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394158 2013-04-03 10:49:55.790526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos latest full 393798 2013-04-03 10:49:55.790543 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394159 update_from_paxos applying incremental log 394159 2013-04-03 10:49:45.787415 mon.0 10.252.24.25:6789/0 258 : [INF] pgmap v29175: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9063B/s wr, 1op/s 2013-04-03 10:49:55.790569 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394159 update_from_paxos applying incremental log 394159 2013-04-03 10:49:54.763814 mon.0 10.252.24.25:6789/0 259 : [INF] pgmap v29176: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2969B/s wr, 0op/s 2013-04-03 10:49:55.790607 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 check_subs 2013-04-03 10:49:55.790643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 create_pending v 394160 2013-04-03 10:49:55.790665 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394159 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:49:55.790730 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos 2013-04-03 10:49:55.790746 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 2013-04-03 10:49:55.790765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:55.797446 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos 2013-04-03 10:49:55.797465 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 2013-04-03 10:49:55.797480 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:55.810652 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos 2013-04-03 10:49:55.810668 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 2013-04-03 10:49:55.810682 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:56.951924 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:49:56.951933 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34ac840 from 10.252.24.88:6789/0 2013-04-03 10:49:56.951989 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 247 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:49:57.440651 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 38 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:49:57.563361 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 2013-04-03 10:49:57.563368 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:49:57.862824 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 2013-04-03 10:49:57.862833 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:49:57.908420 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 2013-04-03 10:49:57.908426 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:49:58.396622 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29176 check_down_pgs 2013-04-03 10:49:58.396656 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29176 v29176: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2969B/s wr, 0op/s 2013-04-03 10:49:58.396684 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:49:58.396707 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:49:58.396744 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:49:58.396774 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos 2013-04-03 10:49:58.396793 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 2013-04-03 10:49:58.396803 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394159 log 2013-04-03 10:49:58.396820 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:58.396839 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:49:58.527686 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 2013-04-03 10:49:58.527693 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:49:58.791287 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos 2013-04-03 10:49:58.791307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 2013-04-03 10:49:58.791325 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:58.791385 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos 2013-04-03 10:49:58.791401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 2013-04-03 10:49:58.791418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:49:59.605474 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140533 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:50:00.306590 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 2013-04-03 10:50:00.306600 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x3598d80 from 10.252.24.88:6789/0 2013-04-03 10:50:00.306651 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 preprocess_query pg_stats(1 pgs tid 65 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:50:00.306675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 prepare_update pg_stats(1 pgs tid 65 v 608) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:50:00.306681 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 prepare_pg_stats pg_stats(1 pgs tid 65 v 608) v1 from osd.2 2013-04-03 10:50:00.306687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 got osd.2 osd_stat(1182 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1182 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) 2013-04-03 10:50:00.356834 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29176 encode_pending v 29177 2013-04-03 10:50:00.406861 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29176 update_from_paxos applying incremental 29177 2013-04-03 10:50:00.406877 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 v29177: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s 2013-04-03 10:50:00.406895 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 send_pg_creates to 0 pgs 2013-04-03 10:50:00.406899 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 update_logger 2013-04-03 10:50:00.406909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 create_pending v 29178 2013-04-03 10:50:00.406918 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29177 _updated_stats for osd.2 10.252.24.30:6801/5656 2013-04-03 10:50:00.406934 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 check_osd_map already seen 608 >= 608 2013-04-03 10:50:00.406936 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 update_logger 2013-04-03 10:50:00.406950 7fcdffcdb700 0 log [INF] : pgmap v29177: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s 2013-04-03 10:50:00.407013 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos 2013-04-03 10:50:00.407032 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 2013-04-03 10:50:00.407050 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:00.407096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos 2013-04-03 10:50:00.407111 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 2013-04-03 10:50:00.407119 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:50:00.407129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:50:00.407152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:50:00.407164 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:50:00.407174 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 logging 2013-04-03 10:50:00.406951 mon.0 10.252.24.25:6789/0 260 : [INF] pgmap v29177: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s 2013-04-03 10:50:00.417292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos 2013-04-03 10:50:00.417310 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 2013-04-03 10:50:00.417326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:00.418936 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos 2013-04-03 10:50:00.418954 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 2013-04-03 10:50:00.418969 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:00.847178 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=608}) v2 2013-04-03 10:50:00.847186 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:50:00.847193 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e608 send_incremental [608..608] to osd.3 10.252.24.88:6804/18376 2013-04-03 10:50:00.847236 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 build_incremental [608..608] 2013-04-03 10:50:00.952010 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:50:00.952018 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34ad080 from 10.252.24.88:6789/0 2013-04-03 10:50:00.952057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 248 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:50:01.398770 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394159 encode_pending v394160 2013-04-03 10:50:01.428844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos 2013-04-03 10:50:01.428873 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394159 2013-04-03 10:50:01.428887 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos latest full 393798 2013-04-03 10:50:01.428904 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394160 update_from_paxos applying incremental log 394160 2013-04-03 10:50:00.406951 mon.0 10.252.24.25:6789/0 260 : [INF] pgmap v29177: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s 2013-04-03 10:50:01.428946 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 check_subs 2013-04-03 10:50:01.428981 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 create_pending v 394161 2013-04-03 10:50:01.429004 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394160 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:50:01.429068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos 2013-04-03 10:50:01.429086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 2013-04-03 10:50:01.429104 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:01.435542 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos 2013-04-03 10:50:01.435558 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 2013-04-03 10:50:01.435572 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:01.440692 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 39 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:50:01.451987 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos 2013-04-03 10:50:01.452004 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 2013-04-03 10:50:01.452019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:01.560916 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=606}) v2 2013-04-03 10:50:01.560923 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:50:01.560930 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e608 send_incremental [606..608] to osd.1 10.252.24.25:6805/3572 2013-04-03 10:50:01.560951 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 build_incremental [606..608] 2013-04-03 10:50:02.433603 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 2013-04-03 10:50:02.433613 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:50:03.396982 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29177 check_down_pgs 2013-04-03 10:50:03.397011 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29177 v29177: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s 2013-04-03 10:50:03.397039 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:50:03.397065 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:50:03.397099 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:50:03.397131 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos 2013-04-03 10:50:03.397149 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 2013-04-03 10:50:03.397157 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394160 log 2013-04-03 10:50:03.397174 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:03.397191 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:50:03.605568 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140534 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:50:04.429687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos 2013-04-03 10:50:04.429707 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 2013-04-03 10:50:04.429725 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:04.429781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos 2013-04-03 10:50:04.429797 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 2013-04-03 10:50:04.429813 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:04.661300 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 preprocess_query pg_stats(1 pgs tid 84 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:50:04.661323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 prepare_update pg_stats(1 pgs tid 84 v 608) v1 from osd.1 10.252.24.25:6805/3572 2013-04-03 10:50:04.661328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 prepare_pg_stats pg_stats(1 pgs tid 84 v 608) v1 from osd.1 2013-04-03 10:50:04.661334 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 got osd.1 osd_stat(1104 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1104 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) 2013-04-03 10:50:04.711491 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29177 encode_pending v 29178 2013-04-03 10:50:04.782225 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29177 update_from_paxos applying incremental 29178 2013-04-03 10:50:04.782245 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 v29178: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s 2013-04-03 10:50:04.782263 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 send_pg_creates to 0 pgs 2013-04-03 10:50:04.782266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 update_logger 2013-04-03 10:50:04.782276 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 create_pending v 29179 2013-04-03 10:50:04.782285 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29178 _updated_stats for osd.1 10.252.24.25:6805/3572 2013-04-03 10:50:04.782301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 check_osd_map already seen 608 >= 608 2013-04-03 10:50:04.782306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 update_logger 2013-04-03 10:50:04.782316 7fcdffcdb700 0 log [INF] : pgmap v29178: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s 2013-04-03 10:50:04.782379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos 2013-04-03 10:50:04.782398 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 2013-04-03 10:50:04.782416 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:04.782459 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos 2013-04-03 10:50:04.782474 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 2013-04-03 10:50:04.782482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:50:04.782492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:50:04.782514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:50:04.782526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 prepare_log log(1 entries) v1 from mon.0 2013-04-03 10:50:04.782535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 logging 2013-04-03 10:50:04.782317 mon.0 10.252.24.25:6789/0 261 : [INF] pgmap v29178: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s 2013-04-03 10:50:04.782884 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos 2013-04-03 10:50:04.782902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 2013-04-03 10:50:04.782919 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:04.807182 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos 2013-04-03 10:50:04.807199 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 2013-04-03 10:50:04.807214 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:04.883195 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 preprocess_query pg_stats(2 pgs tid 417 v 608) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:50:04.883216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 prepare_update pg_stats(2 pgs tid 417 v 608) v1 from osd.0 10.252.58.31:6802/2682 2013-04-03 10:50:04.883221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 prepare_pg_stats pg_stats(2 pgs tid 417 v 608) v1 from osd.0 2013-04-03 10:50:04.883225 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 got osd.0 osd_stat(1146 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1146 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) 2013-04-03 10:50:04.952121 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 2013-04-03 10:50:04.952130 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34acdc0 from 10.252.24.88:6789/0 2013-04-03 10:50:04.952162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 249 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:50:05.440864 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 40 v188) v2 from mds.? 10.252.24.88:6800/18040 2013-04-03 10:50:05.749092 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29178 encode_pending v 29179 2013-04-03 10:50:05.750368 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394160 encode_pending v394161 2013-04-03 10:50:05.778244 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29178 update_from_paxos applying incremental 29179 2013-04-03 10:50:05.778267 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 v29179: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:50:05.778287 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 send_pg_creates to 0 pgs 2013-04-03 10:50:05.778290 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 update_logger 2013-04-03 10:50:05.778300 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 create_pending v 29180 2013-04-03 10:50:05.778309 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29179 _updated_stats for osd.0 10.252.58.31:6802/2682 2013-04-03 10:50:05.778329 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 check_osd_map already seen 608 >= 608 2013-04-03 10:50:05.778333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 update_logger 2013-04-03 10:50:05.778344 7fcdffcdb700 0 log [INF] : pgmap v29179: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:50:05.815506 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos 2013-04-03 10:50:05.815532 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394160 2013-04-03 10:50:05.815546 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos latest full 393798 2013-04-03 10:50:05.815563 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394161 update_from_paxos applying incremental log 394161 2013-04-03 10:50:04.782317 mon.0 10.252.24.25:6789/0 261 : [INF] pgmap v29178: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s 2013-04-03 10:50:05.815609 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 check_subs 2013-04-03 10:50:05.815645 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 create_pending v 394162 2013-04-03 10:50:05.815667 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394161 _updated_log for mon.0 10.252.24.25:6789/0 2013-04-03 10:50:05.815700 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos 2013-04-03 10:50:05.815715 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 2013-04-03 10:50:05.815724 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 2013-04-03 10:50:05.815735 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 preprocess_log log(1 entries) v1 from mon.0 2013-04-03 10:50:05.815751 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 nothing new 2013-04-03 10:50:05.815800 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos 2013-04-03 10:50:05.815816 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 2013-04-03 10:50:05.815832 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:05.816121 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos 2013-04-03 10:50:05.816137 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 2013-04-03 10:50:05.816152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:05.828963 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos 2013-04-03 10:50:05.828982 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 2013-04-03 10:50:05.828997 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:06.859579 7fce026be700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=2 pgs=151512 cs=1 l=0).fault with nothing to send, going to standby 2013-04-03 10:50:06.860059 7fcdfead7700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.25:6802/3144 mds protocol 0 2013-04-03 10:50:06.860036 7fcdfd5c2700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.25:0/5246 client protocol 0 2013-04-03 10:50:06.860168 7fcdfdfcc700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.25:0/1003678 client protocol 0 2013-04-03 10:50:06.860209 7fcdfd3c0700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.25:0/2614196857 client protocol 0 2013-04-03 10:50:06.860269 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:50:06.860272 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: mds.0 10.252.24.25:6802/3144 is open for mds.0 10.252.24.25:6802/3144 2013-04-03 10:50:06.860278 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:50:06.860272 7fcdfdecb700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.30:0/517 client protocol 0 2013-04-03 10:50:06.860318 7fcdfe9d6700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.30:6801/5656 osd protocol 0 2013-04-03 10:50:06.860322 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:06.860342 7fcdfd4c1700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.25:0/2151250553 client protocol 0 2013-04-03 10:50:06.860337 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 35 bytes epoch 5) v1 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:50:06.860350 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=35 2013-04-03 10:50:06.860405 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:50:06.860407 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11042 10.252.24.25:0/2614196857 is open for client.11042 10.252.24.25:0/2614196857 2013-04-03 10:50:06.860411 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:50:06.860432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:06.860446 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11042 10.252.24.25:0/2614196857 2013-04-03 10:50:06.860456 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=30 2013-04-03 10:50:06.860518 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:50:06.860520 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11045 10.252.24.25:0/2151250553 is open for client.11045 10.252.24.25:0/2151250553 2013-04-03 10:50:06.860524 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:50:06.860551 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:06.860579 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11045 10.252.24.25:0/2151250553 2013-04-03 10:50:06.860592 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=30 2013-04-03 10:50:06.860634 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:50:06.860636 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.? 10.252.24.25:0/5246 is open for client.? 10.252.24.25:0/5246 2013-04-03 10:50:06.860640 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:50:06.860658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:06.860671 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.? 10.252.24.25:0/5246 2013-04-03 10:50:06.860683 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=30 2013-04-03 10:50:06.860803 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:50:06.860805 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11802 10.252.24.30:0/517 is open for client.11802 10.252.24.30:0/517 2013-04-03 10:50:06.860809 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:50:06.860830 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:06.860844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11802 10.252.24.30:0/517 2013-04-03 10:50:06.860859 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=30 2013-04-03 10:50:06.860918 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:06.860934 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 144 bytes epoch 0) v1 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:50:06.860944 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=144 2013-04-03 10:50:06.861118 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:50:06.861124 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: osd.2 10.252.24.30:6801/5656 is open for osd.2 10.252.24.30:6801/5656 2013-04-03 10:50:06.861128 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:50:06.861153 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:06.861166 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 26 bytes epoch 5) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:50:06.861179 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=26 2013-04-03 10:50:06.861236 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:06.861267 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.? 10.252.24.25:0/5246 2013-04-03 10:50:06.861279 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 2013-04-03 10:50:06.862240 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:06.862255 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.11045 10.252.24.25:0/2151250553 2013-04-03 10:50:06.862265 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 2013-04-03 10:50:06.862417 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:06.862431 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.11042 10.252.24.25:0/2614196857 2013-04-03 10:50:06.862441 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 2013-04-03 10:50:06.862574 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one 2013-04-03 10:50:06.862576 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11246 10.252.24.25:0/1003678 is open for client.11246 10.252.24.25:0/1003678 2013-04-03 10:50:06.862580 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session 2013-04-03 10:50:06.862601 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:06.862615 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 43 bytes epoch 5) v1 from client.11246 10.252.24.25:0/1003678 2013-04-03 10:50:06.862625 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=43 2013-04-03 10:50:06.862679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:06.862693 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.11802 10.252.24.30:0/517 2013-04-03 10:50:06.862702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 2013-04-03 10:50:06.862840 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+,osdmap=609}) v2 2013-04-03 10:50:06.862847 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:50:06.862884 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:06.862899 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from osd.2 10.252.24.30:6801/5656 2013-04-03 10:50:06.862909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 2013-04-03 10:50:06.863328 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({log-info=394162+,monmap=6+}) v2 2013-04-03 10:50:06.863333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 check_sub client wants log-info ver 394162 2013-04-03 10:50:06.863354 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 check_sub client client.? 10.252.24.25:0/5246 requested version (394162) is greater than ours (394161), which means we already sent him everything we have. 2013-04-03 10:50:06.863366 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:50:06.863385 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 2013-04-03 10:50:06.863389 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:50:06.863404 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 2013-04-03 10:50:06.863408 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:50:06.863443 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:06.863460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 144 bytes epoch 0) v1 from client.11246 10.252.24.25:0/1003678 2013-04-03 10:50:06.863470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=144 2013-04-03 10:50:06.863606 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 2013-04-03 10:50:06.863611 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:50:06.863631 7fcdffcdb700 0 mon.hotcpc9039@0(leader) e5 handle_command mon_command(status v 0) v1 2013-04-03 10:50:06.863880 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=607}) v2 2013-04-03 10:50:06.863886 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:50:06.863891 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e608 send_incremental [607..608] to osd.2 10.252.24.30:6801/5656 2013-04-03 10:50:06.863913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 build_incremental [607..608] 2013-04-03 10:50:06.863984 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=607}) v2 2013-04-03 10:50:06.863990 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:50:06.863994 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e608 send_incremental [607..608] to osd.2 10.252.24.30:6801/5656 2013-04-03 10:50:06.864010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 build_incremental [607..608] 2013-04-03 10:50:06.864198 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osdmap=609}) v2 2013-04-03 10:50:06.864203 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:50:07.605698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140535 v188) v2 from mds.? 10.252.58.31:6800/10908 2013-04-03 10:50:07.696656 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=606}) v2 2013-04-03 10:50:07.696663 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 2013-04-03 10:50:07.696669 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e608 send_incremental [606..608] to osd.0 10.252.58.31:6802/2682 2013-04-03 10:50:07.696690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 build_incremental [606..608] 2013-04-03 10:50:08.397341 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29179 check_down_pgs 2013-04-03 10:50:08.397368 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29179 v29179: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail 2013-04-03 10:50:08.397396 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:50:08.397418 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:50:08.397455 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 2013-04-03 10:50:08.397486 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos 2013-04-03 10:50:08.397505 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 2013-04-03 10:50:08.397514 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 log 2013-04-03 10:50:08.397535 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:08.397554 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:50:08.815907 7fce027bf700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).fault 2013-04-03 10:50:08.816389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos 2013-04-03 10:50:08.816409 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 2013-04-03 10:50:08.816428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:08.862340 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 preprocess_query pg_stats(1 pgs tid 7 v 608) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:50:08.862362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 prepare_update pg_stats(1 pgs tid 7 v 608) v1 from osd.3 10.252.24.88:6804/18376 2013-04-03 10:50:08.862367 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 prepare_pg_stats pg_stats(1 pgs tid 7 v 608) v1 from osd.3 2013-04-03 10:50:08.862373 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 got osd.3 osd_stat(1182 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) 2013-04-03 10:50:08.912574 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29179 encode_pending v 29180 2013-04-03 10:50:08.951428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 250 v188) v2 from mds.0 10.252.24.25:6802/3144 2013-04-03 10:50:11.609209 7fce026be700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3657780 sd=38 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33225/0) 2013-04-03 10:50:13.397706 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:50:13.397742 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:50:13.397764 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos 2013-04-03 10:50:13.397783 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 2013-04-03 10:50:13.397791 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 log 2013-04-03 10:50:13.397809 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:13.397828 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:50:18.397970 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:50:18.398012 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in 2013-04-03 10:50:18.398034 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos 2013-04-03 10:50:18.398066 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 2013-04-03 10:50:18.398075 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 log 2013-04-03 10:50:18.398093 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos 2013-04-03 10:50:18.398112 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth 2013-04-03 10:50:18.816470 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:50:18.816482 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 2013-04-03 10:50:18.907166 7fcdffcdb700 5 mon.hotcpc9039@0(leader).elector(356) handle_propose from mon.1 2013-04-03 10:50:18.907170 7fcdffcdb700 10 mon.hotcpc9039@0(leader).elector(356) bump_epoch 356 to 357 2013-04-03 10:50:18.908377 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 reset 2013-04-03 10:50:18.908392 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish 2013-04-03 10:50:18.908465 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 start_election 2013-04-03 10:50:18.908470 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 cancel_probe_timeout (none scheduled) 2013-04-03 10:50:18.908476 7fcdffcdb700 0 log [INF] : mon.hotcpc9039 calling new monitor election 2013-04-03 10:50:18.908497 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(357) start -- can i be leader? 2013-04-03 10:50:18.908512 7fcdffcdb700 1 mon.hotcpc9039@0(electing).elector(357) init, last seen epoch 357 2013-04-03 10:50:18.909228 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(357) handle_ack from mon.1 2013-04-03 10:50:18.909231 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(357) so far i have {0=4294967295,1=4294967295} 2013-04-03 10:50:23.908610 7fce004dc700 5 mon.hotcpc9039@0(electing).elector(357) election timer expired 2013-04-03 10:50:23.908627 7fce004dc700 10 mon.hotcpc9039@0(electing).elector(357) bump_epoch 357 to 358 2013-04-03 10:50:23.908956 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 win_election, epoch 358 quorum is 0,1 features are 4294967295 2013-04-03 10:50:23.908966 7fce004dc700 0 log [INF] : mon.hotcpc9039@0 won leader election with quorum 0,1 2013-04-03 10:50:23.909494 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish 2013-04-03 10:50:23.909497 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 resend_routed_requests 2013-04-03 10:50:23.909499 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 register_cluster_logger - already registered 2013-04-03 10:50:23.909501 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start 2013-04-03 10:50:23.909502 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round curr 0 2013-04-03 10:50:23.909504 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round new 1 2013-04-03 10:50:23.909505 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck 2013-04-03 10:50:23.909506 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck start timecheck epoch 358 round 1 2013-04-03 10:50:23.909510 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck send time_check( ping e 358 r 1 ) v1 to mon.1 10.252.24.30:6789/0 2013-04-03 10:50:23.909523 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round setting up next event 2013-04-03 10:50:23.979493 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29179 update_from_paxos applying incremental 29180 2013-04-03 10:50:23.979512 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29180 v29180: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 243B/s wr, 0op/s 2013-04-03 10:50:23.979530 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29180 send_pg_creates to 0 pgs 2013-04-03 10:50:23.979533 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29180 update_logger 2013-04-03 10:50:23.979543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29180 create_pending v 29181 2013-04-03 10:50:23.979551 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29180 check_osd_map already seen 608 >= 608 2013-04-03 10:50:23.979553 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29180 update_logger 2013-04-03 10:50:23.979562 7fcdffcdb700 0 log [INF] : pgmap v29180: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 243B/s wr, 0op/s 2013-04-03 10:50:23.979605 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 create_pending e189 2013-04-03 10:50:23.979625 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby 2013-04-03 10:50:23.979634 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 no beacon from 10187 10.252.58.31:6800/10908 mds.-1.0 up:standby since 2013-04-03 10:50:07.605709 2013-04-03 10:50:23.979644 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 removing 10187 10.252.58.31:6800/10908 mds.-1.0 up:standby (laggy) 2013-04-03 10:50:23.979650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 no beacon from 11234 10.252.24.25:6802/3144 mds.0.29 up:active since 2013-04-03 10:50:08.951439 2013-04-03 10:50:23.979654 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 replacing 11234 10.252.24.25:6802/3144 mds.0.29 up:active with 11904/hotcpc12533 10.252.24.88:6800/18040 2013-04-03 10:50:23.979659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 blacklist 10.252.24.25:6802/3144 until 2013-04-03 11:14:23.979632 2013-04-03 10:50:23.979664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 no beacon from 11904 10.252.24.88:6800/18040 mds.0.30 up:replay since 2013-04-03 10:50:05.440875 2013-04-03 10:50:23.979668 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 marking 11904 10.252.24.88:6800/18040 mds.0.30 up:replay laggy 2013-04-03 10:50:23.980641 7fcdffcdb700 -1 mon/PaxosService.cc: In function 'void PaxosService::propose_pending()' thread 7fcdffcdb700 time 2013-04-03 10:50:23.979673 mon/PaxosService.cc: 127: FAILED assert(have_pending) ceph version 0.59 (cbae6a435c62899f857775f66659de052fb0e759) 1: (PaxosService::propose_pending()+0xbe3) [0x4d7043] 2: (MDSMonitor::tick()+0x2dce) [0x5127ee] 3: (MDSMonitor::on_active()+0x1f) [0x50d89f] 4: (PaxosService::_active()+0x2bf) [0x4d743f] 5: (Context::complete(int)+0xa) [0x4ac1da] 6: (finish_contexts(CephContext*, std::list >&, int)+0x106) [0x4af196] 7: (Paxos::handle_last(MMonPaxos*)+0x9d0) [0x4d03c0] 8: (Paxos::dispatch(PaxosServiceMessage*)+0x27b) [0x4d135b] 9: (Monitor::_ms_dispatch(Message*)+0x136a) [0x4ab30a] 10: (Monitor::ms_dispatch(Message*)+0x32) [0x4c40e2] 11: (DispatchQueue::entry()+0x311) [0x686d11] 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x682aad] 13: (()+0x68ca) [0x7fce057df8ca] 14: (clone()+0x6d) [0x7fce044d686d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -25456> 2013-04-03 10:33:48.184374 7fce0612e780 5 asok(0x1b9d000) register_command perfcounters_dump hook 0x1b91010 -25455> 2013-04-03 10:33:48.184393 7fce0612e780 5 asok(0x1b9d000) register_command 1 hook 0x1b91010 -25454> 2013-04-03 10:33:48.184395 7fce0612e780 5 asok(0x1b9d000) register_command perf dump hook 0x1b91010 -25453> 2013-04-03 10:33:48.184402 7fce0612e780 5 asok(0x1b9d000) register_command perfcounters_schema hook 0x1b91010 -25452> 2013-04-03 10:33:48.184405 7fce0612e780 5 asok(0x1b9d000) register_command 2 hook 0x1b91010 -25451> 2013-04-03 10:33:48.184408 7fce0612e780 5 asok(0x1b9d000) register_command perf schema hook 0x1b91010 -25450> 2013-04-03 10:33:48.184410 7fce0612e780 5 asok(0x1b9d000) register_command config show hook 0x1b91010 -25449> 2013-04-03 10:33:48.184413 7fce0612e780 5 asok(0x1b9d000) register_command config set hook 0x1b91010 -25448> 2013-04-03 10:33:48.184416 7fce0612e780 5 asok(0x1b9d000) register_command log flush hook 0x1b91010 -25447> 2013-04-03 10:33:48.184418 7fce0612e780 5 asok(0x1b9d000) register_command log dump hook 0x1b91010 -25446> 2013-04-03 10:33:48.184421 7fce0612e780 5 asok(0x1b9d000) register_command log reopen hook 0x1b91010 -25445> 2013-04-03 10:33:48.185727 7fce0612e780 0 ceph version 0.59 (cbae6a435c62899f857775f66659de052fb0e759), process ceph-mon, pid 4155 -25444> 2013-04-03 10:33:48.185736 7fce0612e780 10 needs_conversion -25443> 2013-04-03 10:33:48.230591 7fce0612e780 1 -- 10.252.24.25:6789/0 learned my addr 10.252.24.25:6789/0 -25442> 2013-04-03 10:33:48.230602 7fce0612e780 1 accepter.accepter.bind my_inst.addr is 10.252.24.25:6789/0 need_addr=0 -25441> 2013-04-03 10:33:48.230611 7fce0612e780 5 adding auth protocol: cephx -25440> 2013-04-03 10:33:48.230614 7fce0612e780 5 adding auth protocol: cephx -25439> 2013-04-03 10:33:48.230638 7fce0612e780 1 mon.hotcpc9039@-1(probing) e3 preinit fsid d6e91ff1-9b43-4972-b7a7-a22d84497d78 -25438> 2013-04-03 10:33:48.230696 7fce0612e780 10 mon.hotcpc9039@-1(probing) e3 check_fsid cluster_uuid contains 'd6e91ff1-9b43-4972-b7a7-a22d84497d78' -25437> 2013-04-03 10:33:48.230720 7fce0612e780 10 mon.hotcpc9039@-1(probing) e3 features compat={},rocompat={},incompat={1=initial feature set (~v.18),3=single paxos with k/v store (v0.?)} -25436> 2013-04-03 10:33:48.230746 7fce0612e780 10 mon.hotcpc9039@-1(probing) e3 has_ever_joined = 1 -25435> 2013-04-03 10:33:48.230785 7fce0612e780 10 mon.hotcpc9039@-1(probing) e3 init_paxos -25434> 2013-04-03 10:33:48.231504 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v0 update_from_paxos loading latest full pgmap v28488 -25433> 2013-04-03 10:33:48.233074 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28488 update_from_paxos applying incremental 28489 -25432> 2013-04-03 10:33:48.233097 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28489 v28489: 240 pgs: 240 active+clean; 233 MB data, 5621 MB used, 1067 GB / 1079 GB avail; 2051B/s wr, 0op/s -25431> 2013-04-03 10:33:48.233138 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28489 update_from_paxos applying incremental 28490 -25430> 2013-04-03 10:33:48.233143 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28490 v28490: 240 pgs: 240 active+clean; 233 MB data, 5621 MB used, 1067 GB / 1079 GB avail; 0B/s wr, 0op/s -25429> 2013-04-03 10:33:48.233712 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28490 update_from_paxos applying incremental 28491 -25428> 2013-04-03 10:33:48.234089 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28491 v28491: 240 pgs: 73 active+clean, 167 stale+active+clean; 246 MB data, 5627 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 283B/s wr, 0op/s -25427> 2013-04-03 10:33:48.234227 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28491 update_from_paxos applying incremental 28492 -25426> 2013-04-03 10:33:48.234234 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28492 v28492: 240 pgs: 73 active+clean, 167 stale+active+clean; 246 MB data, 5627 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 6439KB/s wr, 264op/s -25425> 2013-04-03 10:33:48.234296 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28492 update_from_paxos applying incremental 28493 -25424> 2013-04-03 10:33:48.234656 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28493 v28493: 240 pgs: 73 active+clean, 165 peering, 2 active+recovering; 246 MB data, 5638 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 0B/s wr, 195op/s; 2/7390 degraded (0.027%); recovering 0 o/s, 0B/s -25423> 2013-04-03 10:33:48.234745 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28493 update_from_paxos applying incremental 28494 -25422> 2013-04-03 10:33:48.234752 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28494 v28494: 240 pgs: 73 active+clean, 165 peering, 2 active+recovering; 246 MB data, 5638 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 0B/s wr, 306op/s; 2/7390 degraded (0.027%); recovering 1 o/s, 0B/s -25421> 2013-04-03 10:33:48.234789 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28494 update_from_paxos applying incremental 28495 -25420> 2013-04-03 10:33:48.234892 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28495 v28495: 240 pgs: 108 active+clean, 10 active+clean+replay, 120 peering, 2 active+recovering; 246 MB data, 5641 MB used, 1067 GB / 1079 GB avail; 2/7390 degraded (0.027%) -25419> 2013-04-03 10:33:48.235161 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28495 update_from_paxos applying incremental 28496 -25418> 2013-04-03 10:33:48.235580 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28496 v28496: 240 pgs: 215 active+clean, 25 active+clean+replay; 246 MB data, 5655 MB used, 1067 GB / 1079 GB avail; recovering 0 o/s, 0B/s -25417> 2013-04-03 10:33:48.235644 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28496 update_from_paxos applying incremental 28497 -25416> 2013-04-03 10:33:48.235665 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28497 v28497: 240 pgs: 215 active+clean, 25 active+clean+replay; 246 MB data, 5655 MB used, 1067 GB / 1079 GB avail; recovering 0 o/s, 0B/s -25415> 2013-04-03 10:33:48.235696 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28497 update_from_paxos applying incremental 28498 -25414> 2013-04-03 10:33:48.235733 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28498 v28498: 240 pgs: 215 active+clean, 25 active+clean+replay; 246 MB data, 5658 MB used, 1067 GB / 1079 GB avail -25413> 2013-04-03 10:33:48.235983 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28498 update_from_paxos applying incremental 28499 -25412> 2013-04-03 10:33:48.236074 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28499 v28499: 240 pgs: 240 active+clean; 246 MB data, 5646 MB used, 1067 GB / 1079 GB avail; 39B/s wr, 0op/s -25411> 2013-04-03 10:33:48.236209 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28499 update_from_paxos applying incremental 28500 -25410> 2013-04-03 10:33:48.236254 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28500 v28500: 240 pgs: 240 active+clean; 241 MB data, 5646 MB used, 1067 GB / 1079 GB avail; 39B/s wr, 0op/s -25409> 2013-04-03 10:33:48.236305 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28500 update_from_paxos applying incremental 28501 -25408> 2013-04-03 10:33:48.236381 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28501 v28501: 240 pgs: 240 active+clean; 235 MB data, 5640 MB used, 1067 GB / 1079 GB avail; 25110B/s wr, 7op/s -25407> 2013-04-03 10:33:48.236423 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28501 update_from_paxos applying incremental 28502 -25406> 2013-04-03 10:33:48.236462 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28502 v28502: 240 pgs: 240 active+clean; 233 MB data, 5641 MB used, 1067 GB / 1079 GB avail; 74616B/s wr, 25op/s -25405> 2013-04-03 10:33:48.236677 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28502 update_from_paxos applying incremental 28503 -25404> 2013-04-03 10:33:48.236697 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28503 v28503: 240 pgs: 240 active+clean; 233 MB data, 5633 MB used, 1067 GB / 1079 GB avail; 361B/s rd, 27104B/s wr, 15op/s -25403> 2013-04-03 10:33:48.236726 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28503 update_from_paxos applying incremental 28504 -25402> 2013-04-03 10:33:48.236734 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28504 v28504: 240 pgs: 240 active+clean; 233 MB data, 5629 MB used, 1067 GB / 1079 GB avail; 314B/s rd, 92494B/s wr, 39op/s -25401> 2013-04-03 10:33:48.236760 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28504 update_from_paxos applying incremental 28505 -25400> 2013-04-03 10:33:48.236773 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28505 v28505: 240 pgs: 240 active+clean; 233 MB data, 5629 MB used, 1067 GB / 1079 GB avail; 74453B/s wr, 35op/s -25399> 2013-04-03 10:33:48.236799 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28505 update_from_paxos applying incremental 28506 -25398> 2013-04-03 10:33:48.236808 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28506 v28506: 240 pgs: 240 active+clean; 233 MB data, 5629 MB used, 1067 GB / 1079 GB avail; 5443B/s wr, 1op/s -25397> 2013-04-03 10:33:48.236834 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28506 update_from_paxos applying incremental 28507 -25396> 2013-04-03 10:33:48.236842 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28507 v28507: 240 pgs: 240 active+clean; 233 MB data, 5630 MB used, 1067 GB / 1079 GB avail; 3639B/s wr, 0op/s -25395> 2013-04-03 10:33:48.236877 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28507 update_from_paxos applying incremental 28508 -25394> 2013-04-03 10:33:48.236888 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28508 v28508: 240 pgs: 240 active+clean; 229 MB data, 5630 MB used, 1067 GB / 1079 GB avail; 332B/s wr, 0op/s -25393> 2013-04-03 10:33:48.236914 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28508 update_from_paxos applying incremental 28509 -25392> 2013-04-03 10:33:48.236923 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28509 v28509: 240 pgs: 240 active+clean; 229 MB data, 5630 MB used, 1067 GB / 1079 GB avail; 19124B/s wr, 6op/s -25391> 2013-04-03 10:33:48.236948 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28509 update_from_paxos applying incremental 28510 -25390> 2013-04-03 10:33:48.236956 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28510 v28510: 240 pgs: 240 active+clean; 229 MB data, 5630 MB used, 1067 GB / 1079 GB avail; 4769B/s wr, 1op/s -25389> 2013-04-03 10:33:48.236982 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28510 update_from_paxos applying incremental 28511 -25388> 2013-04-03 10:33:48.236991 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28511 v28511: 240 pgs: 240 active+clean; 229 MB data, 5620 MB used, 1067 GB / 1079 GB avail; 594B/s wr, 0op/s -25387> 2013-04-03 10:33:48.237019 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28511 update_from_paxos applying incremental 28512 -25386> 2013-04-03 10:33:48.237028 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28512 v28512: 240 pgs: 240 active+clean; 229 MB data, 5620 MB used, 1067 GB / 1079 GB avail; 2662B/s wr, 0op/s -25385> 2013-04-03 10:33:48.237054 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28512 update_from_paxos applying incremental 28513 -25384> 2013-04-03 10:33:48.237062 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28513 v28513: 240 pgs: 240 active+clean; 229 MB data, 5620 MB used, 1067 GB / 1079 GB avail; 8403B/s wr, 1op/s -25383> 2013-04-03 10:33:48.237087 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28513 update_from_paxos applying incremental 28514 -25382> 2013-04-03 10:33:48.237093 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28514 v28514: 240 pgs: 240 active+clean; 229 MB data, 5617 MB used, 1067 GB / 1079 GB avail; 5433B/s wr, 0op/s -25381> 2013-04-03 10:33:48.237118 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28514 update_from_paxos applying incremental 28515 -25380> 2013-04-03 10:33:48.237126 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28515 v28515: 240 pgs: 240 active+clean; 231 MB data, 5619 MB used, 1067 GB / 1079 GB avail; 589KB/s wr, 0op/s -25379> 2013-04-03 10:33:48.237150 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28515 update_from_paxos applying incremental 28516 -25378> 2013-04-03 10:33:48.237158 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28516 v28516: 240 pgs: 240 active+clean; 231 MB data, 5618 MB used, 1067 GB / 1079 GB avail; 607KB/s wr, 0op/s -25377> 2013-04-03 10:33:48.237192 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28516 update_from_paxos applying incremental 28517 -25376> 2013-04-03 10:33:48.237200 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28517 v28517: 240 pgs: 240 active+clean; 231 MB data, 5618 MB used, 1067 GB / 1079 GB avail; 780B/s wr, 0op/s -25375> 2013-04-03 10:33:48.237226 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28517 update_from_paxos applying incremental 28518 -25374> 2013-04-03 10:33:48.237235 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28518 v28518: 240 pgs: 240 active+clean; 231 MB data, 5618 MB used, 1067 GB / 1079 GB avail; 254B/s wr, 0op/s -25373> 2013-04-03 10:33:48.237260 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28518 update_from_paxos applying incremental 28519 -25372> 2013-04-03 10:33:48.237268 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28519 v28519: 240 pgs: 240 active+clean; 231 MB data, 5618 MB used, 1067 GB / 1079 GB avail; 31B/s wr, 0op/s -25371> 2013-04-03 10:33:48.237294 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28519 update_from_paxos applying incremental 28520 -25370> 2013-04-03 10:33:48.237302 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28520 v28520: 240 pgs: 240 active+clean; 231 MB data, 5612 MB used, 1067 GB / 1079 GB avail -25369> 2013-04-03 10:33:48.237335 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28520 update_from_paxos applying incremental 28521 -25368> 2013-04-03 10:33:48.237343 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28521 v28521: 240 pgs: 240 active+clean; 231 MB data, 5612 MB used, 1067 GB / 1079 GB avail; 533B/s wr, 0op/s -25367> 2013-04-03 10:33:48.237369 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28521 update_from_paxos applying incremental 28522 -25366> 2013-04-03 10:33:48.237377 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28522 v28522: 240 pgs: 240 active+clean; 231 MB data, 5612 MB used, 1067 GB / 1079 GB avail; 579B/s wr, 0op/s -25365> 2013-04-03 10:33:48.237403 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28522 update_from_paxos applying incremental 28523 -25364> 2013-04-03 10:33:48.237413 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28523 v28523: 240 pgs: 240 active+clean; 231 MB data, 5612 MB used, 1067 GB / 1079 GB avail; 82559B/s wr, 0op/s -25363> 2013-04-03 10:33:48.237440 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28523 update_from_paxos applying incremental 28524 -25362> 2013-04-03 10:33:48.237448 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28524 v28524: 240 pgs: 240 active+clean; 231 MB data, 5612 MB used, 1067 GB / 1079 GB avail; 115KB/s wr, 0op/s -25361> 2013-04-03 10:33:48.237479 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28524 update_from_paxos applying incremental 28525 -25360> 2013-04-03 10:33:48.237488 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28525 v28525: 240 pgs: 240 active+clean; 231 MB data, 5612 MB used, 1067 GB / 1079 GB avail; 1456B/s wr, 0op/s -25359> 2013-04-03 10:33:48.237513 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28525 update_from_paxos applying incremental 28526 -25358> 2013-04-03 10:33:48.237521 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28526 v28526: 240 pgs: 240 active+clean; 231 MB data, 5612 MB used, 1067 GB / 1079 GB avail; 1011B/s wr, 0op/s -25357> 2013-04-03 10:33:48.237546 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28526 update_from_paxos applying incremental 28527 -25356> 2013-04-03 10:33:48.237555 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28527 v28527: 240 pgs: 240 active+clean; 231 MB data, 5608 MB used, 1067 GB / 1079 GB avail; 664B/s wr, 0op/s -25355> 2013-04-03 10:33:48.237581 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28527 update_from_paxos applying incremental 28528 -25354> 2013-04-03 10:33:48.237590 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28528 v28528: 240 pgs: 240 active+clean; 231 MB data, 5608 MB used, 1067 GB / 1079 GB avail; 571B/s wr, 0op/s -25353> 2013-04-03 10:33:48.237615 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28528 update_from_paxos applying incremental 28529 -25352> 2013-04-03 10:33:48.237623 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28529 v28529: 240 pgs: 240 active+clean; 231 MB data, 5608 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -25351> 2013-04-03 10:33:48.237649 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28529 update_from_paxos applying incremental 28530 -25350> 2013-04-03 10:33:48.237657 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28530 v28530: 240 pgs: 240 active+clean; 231 MB data, 5608 MB used, 1067 GB / 1079 GB avail; 602B/s wr, 0op/s -25349> 2013-04-03 10:33:48.237684 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28530 update_from_paxos applying incremental 28531 -25348> 2013-04-03 10:33:48.237692 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28531 v28531: 240 pgs: 240 active+clean; 231 MB data, 5608 MB used, 1067 GB / 1079 GB avail; 646B/s wr, 0op/s -25347> 2013-04-03 10:33:48.237718 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28531 update_from_paxos applying incremental 28532 -25346> 2013-04-03 10:33:48.237726 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28532 v28532: 240 pgs: 240 active+clean; 231 MB data, 5608 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -25345> 2013-04-03 10:33:48.237859 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28532 update_from_paxos applying incremental 28533 -25344> 2013-04-03 10:33:48.237873 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28533 v28533: 240 pgs: 240 active+clean; 231 MB data, 5608 MB used, 1067 GB / 1079 GB avail; 963B/s wr, 0op/s -25343> 2013-04-03 10:33:48.237901 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28533 update_from_paxos applying incremental 28534 -25342> 2013-04-03 10:33:48.237910 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28534 v28534: 240 pgs: 240 active+clean; 231 MB data, 5608 MB used, 1067 GB / 1079 GB avail; 1157B/s wr, 0op/s -25341> 2013-04-03 10:33:48.237936 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28534 update_from_paxos applying incremental 28535 -25340> 2013-04-03 10:33:48.237944 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28535 v28535: 240 pgs: 240 active+clean; 231 MB data, 5608 MB used, 1067 GB / 1079 GB avail; 1236B/s wr, 0op/s -25339> 2013-04-03 10:33:48.237970 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28535 update_from_paxos applying incremental 28536 -25338> 2013-04-03 10:33:48.237979 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28536 v28536: 240 pgs: 240 active+clean; 231 MB data, 5608 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s -25337> 2013-04-03 10:33:48.238004 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28536 update_from_paxos applying incremental 28537 -25336> 2013-04-03 10:33:48.238011 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28537 v28537: 240 pgs: 240 active+clean; 231 MB data, 5610 MB used, 1067 GB / 1079 GB avail; 923B/s wr, 0op/s -25335> 2013-04-03 10:33:48.238036 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28537 update_from_paxos applying incremental 28538 -25334> 2013-04-03 10:33:48.238041 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28538 v28538: 240 pgs: 240 active+clean; 231 MB data, 5610 MB used, 1067 GB / 1079 GB avail -25333> 2013-04-03 10:33:48.238066 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28538 update_from_paxos applying incremental 28539 -25332> 2013-04-03 10:33:48.238074 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28539 v28539: 240 pgs: 240 active+clean; 231 MB data, 5610 MB used, 1067 GB / 1079 GB avail; 637B/s wr, 0op/s -25331> 2013-04-03 10:33:48.238100 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28539 update_from_paxos applying incremental 28540 -25330> 2013-04-03 10:33:48.238108 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28540 v28540: 240 pgs: 240 active+clean; 231 MB data, 5610 MB used, 1067 GB / 1079 GB avail; 532B/s wr, 0op/s -25329> 2013-04-03 10:33:48.238133 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28540 update_from_paxos applying incremental 28541 -25328> 2013-04-03 10:33:48.238141 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28541 v28541: 240 pgs: 240 active+clean; 231 MB data, 5610 MB used, 1067 GB / 1079 GB avail; 496B/s wr, 0op/s -25327> 2013-04-03 10:33:48.238166 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28541 update_from_paxos applying incremental 28542 -25326> 2013-04-03 10:33:48.238174 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28542 v28542: 240 pgs: 240 active+clean; 231 MB data, 5610 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -25325> 2013-04-03 10:33:48.238206 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28542 update_from_paxos applying incremental 28543 -25324> 2013-04-03 10:33:48.238215 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28543 v28543: 240 pgs: 240 active+clean; 237 MB data, 5610 MB used, 1067 GB / 1079 GB avail; 335KB/s wr, 0op/s -25323> 2013-04-03 10:33:48.238241 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28543 update_from_paxos applying incremental 28544 -25322> 2013-04-03 10:33:48.238249 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28544 v28544: 240 pgs: 240 active+clean; 237 MB data, 5611 MB used, 1067 GB / 1079 GB avail; 273KB/s wr, 0op/s -25321> 2013-04-03 10:33:48.238274 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28544 update_from_paxos applying incremental 28545 -25320> 2013-04-03 10:33:48.238282 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28545 v28545: 240 pgs: 240 active+clean; 237 MB data, 5611 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -25319> 2013-04-03 10:33:48.238308 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28545 update_from_paxos applying incremental 28546 -25318> 2013-04-03 10:33:48.238317 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28546 v28546: 240 pgs: 240 active+clean; 237 MB data, 5617 MB used, 1067 GB / 1079 GB avail; 24573B/s wr, 0op/s -25317> 2013-04-03 10:33:48.238343 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28546 update_from_paxos applying incremental 28547 -25316> 2013-04-03 10:33:48.238352 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28547 v28547: 240 pgs: 240 active+clean; 237 MB data, 5617 MB used, 1067 GB / 1079 GB avail; 14402B/s wr, 0op/s -25315> 2013-04-03 10:33:48.238377 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28547 update_from_paxos applying incremental 28548 -25314> 2013-04-03 10:33:48.238386 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28548 v28548: 240 pgs: 240 active+clean; 237 MB data, 5617 MB used, 1067 GB / 1079 GB avail; 4058B/s wr, 0op/s -25313> 2013-04-03 10:33:48.238412 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28548 update_from_paxos applying incremental 28549 -25312> 2013-04-03 10:33:48.238421 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28549 v28549: 240 pgs: 240 active+clean; 237 MB data, 5617 MB used, 1067 GB / 1079 GB avail; 7582B/s wr, 0op/s -25311> 2013-04-03 10:33:48.238447 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28549 update_from_paxos applying incremental 28550 -25310> 2013-04-03 10:33:48.238456 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28550 v28550: 240 pgs: 240 active+clean; 237 MB data, 5617 MB used, 1067 GB / 1079 GB avail; 7456B/s wr, 0op/s -25309> 2013-04-03 10:33:48.238546 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28550 update_from_paxos applying incremental 28551 -25308> 2013-04-03 10:33:48.238557 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28551 v28551: 240 pgs: 240 active+clean; 237 MB data, 5617 MB used, 1067 GB / 1079 GB avail; 1194B/s wr, 0op/s -25307> 2013-04-03 10:33:48.238586 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28551 update_from_paxos applying incremental 28552 -25306> 2013-04-03 10:33:48.238595 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28552 v28552: 240 pgs: 240 active+clean; 237 MB data, 5617 MB used, 1067 GB / 1079 GB avail; 1175B/s wr, 0op/s -25305> 2013-04-03 10:33:48.238622 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28552 update_from_paxos applying incremental 28553 -25304> 2013-04-03 10:33:48.238631 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28553 v28553: 240 pgs: 240 active+clean; 237 MB data, 5617 MB used, 1067 GB / 1079 GB avail; 11468B/s wr, 0op/s -25303> 2013-04-03 10:33:48.238658 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28553 update_from_paxos applying incremental 28554 -25302> 2013-04-03 10:33:48.238666 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28554 v28554: 240 pgs: 240 active+clean; 237 MB data, 5617 MB used, 1067 GB / 1079 GB avail; 6552B/s wr, 0op/s -25301> 2013-04-03 10:33:48.238693 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28554 update_from_paxos applying incremental 28555 -25300> 2013-04-03 10:33:48.238701 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28555 v28555: 240 pgs: 240 active+clean; 238 MB data, 5617 MB used, 1067 GB / 1079 GB avail; 5733B/s wr, 0op/s -25299> 2013-04-03 10:33:48.238726 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28555 update_from_paxos applying incremental 28556 -25298> 2013-04-03 10:33:48.238736 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28556 v28556: 240 pgs: 240 active+clean; 238 MB data, 5617 MB used, 1067 GB / 1079 GB avail; 5733B/s wr, 0op/s -25297> 2013-04-03 10:33:48.238762 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28556 update_from_paxos applying incremental 28557 -25296> 2013-04-03 10:33:48.238770 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28557 v28557: 240 pgs: 240 active+clean; 238 MB data, 5617 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -25295> 2013-04-03 10:33:48.238796 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28557 update_from_paxos applying incremental 28558 -25294> 2013-04-03 10:33:48.238800 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28558 v28558: 240 pgs: 240 active+clean; 238 MB data, 5617 MB used, 1067 GB / 1079 GB avail; 95B/s wr, 0op/s -25293> 2013-04-03 10:33:48.238832 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28558 update_from_paxos applying incremental 28559 -25292> 2013-04-03 10:33:48.239101 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28559 v28559: 240 pgs: 240 active+clean; 238 MB data, 5625 MB used, 1067 GB / 1079 GB avail -25291> 2013-04-03 10:33:48.239182 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28559 update_from_paxos applying incremental 28560 -25290> 2013-04-03 10:33:48.239340 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28560 v28560: 240 pgs: 240 active+clean; 238 MB data, 5625 MB used, 1067 GB / 1079 GB avail; 34201B/s wr, 0op/s -25289> 2013-04-03 10:33:48.239400 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28560 update_from_paxos applying incremental 28561 -25288> 2013-04-03 10:33:48.239502 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28561 v28561: 240 pgs: 240 active+clean; 238 MB data, 5625 MB used, 1067 GB / 1079 GB avail; 33702B/s wr, 0op/s -25287> 2013-04-03 10:33:48.239539 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28561 update_from_paxos applying incremental 28562 -25286> 2013-04-03 10:33:48.239548 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28562 v28562: 240 pgs: 240 active+clean; 238 MB data, 5625 MB used, 1067 GB / 1079 GB avail; 661B/s wr, 0op/s -25285> 2013-04-03 10:33:48.239806 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28562 update_from_paxos applying incremental 28563 -25284> 2013-04-03 10:33:48.239820 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28563 v28563: 240 pgs: 240 active+clean; 238 MB data, 5625 MB used, 1067 GB / 1079 GB avail; 801B/s wr, 0op/s -25283> 2013-04-03 10:33:48.239851 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28563 update_from_paxos applying incremental 28564 -25282> 2013-04-03 10:33:48.239860 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28564 v28564: 240 pgs: 240 active+clean; 238 MB data, 5625 MB used, 1067 GB / 1079 GB avail; 12360B/s wr, 0op/s -25281> 2013-04-03 10:33:48.239886 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28564 update_from_paxos applying incremental 28565 -25280> 2013-04-03 10:33:48.239895 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28565 v28565: 240 pgs: 240 active+clean; 238 MB data, 5625 MB used, 1067 GB / 1079 GB avail; 7467B/s wr, 0op/s -25279> 2013-04-03 10:33:48.239921 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28565 update_from_paxos applying incremental 28566 -25278> 2013-04-03 10:33:48.239929 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28566 v28566: 240 pgs: 240 active+clean; 238 MB data, 5625 MB used, 1067 GB / 1079 GB avail; 11316B/s wr, 0op/s -25277> 2013-04-03 10:33:48.239957 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28566 update_from_paxos applying incremental 28567 -25276> 2013-04-03 10:33:48.239966 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28567 v28567: 240 pgs: 240 active+clean; 238 MB data, 5625 MB used, 1067 GB / 1079 GB avail; 33055B/s wr, 0op/s -25275> 2013-04-03 10:33:48.239993 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28567 update_from_paxos applying incremental 28568 -25274> 2013-04-03 10:33:48.240001 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28568 v28568: 240 pgs: 240 active+clean; 238 MB data, 5625 MB used, 1067 GB / 1079 GB avail; 41618B/s wr, 0op/s -25273> 2013-04-03 10:33:48.240026 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28568 update_from_paxos applying incremental 28569 -25272> 2013-04-03 10:33:48.240034 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28569 v28569: 240 pgs: 240 active+clean; 238 MB data, 5625 MB used, 1067 GB / 1079 GB avail; 1237B/s wr, 0op/s -25271> 2013-04-03 10:33:48.240067 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28569 update_from_paxos applying incremental 28570 -25270> 2013-04-03 10:33:48.240075 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28570 v28570: 240 pgs: 240 active+clean; 238 MB data, 5625 MB used, 1067 GB / 1079 GB avail; 1160B/s wr, 0op/s -25269> 2013-04-03 10:33:48.240101 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28570 update_from_paxos applying incremental 28571 -25268> 2013-04-03 10:33:48.240109 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28571 v28571: 240 pgs: 240 active+clean; 239 MB data, 5626 MB used, 1067 GB / 1079 GB avail; 31671B/s wr, 0op/s -25267> 2013-04-03 10:33:48.240134 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28571 update_from_paxos applying incremental 28572 -25266> 2013-04-03 10:33:48.240142 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28572 v28572: 240 pgs: 240 active+clean; 239 MB data, 5626 MB used, 1067 GB / 1079 GB avail; 33055B/s wr, 0op/s -25265> 2013-04-03 10:33:48.240167 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28572 update_from_paxos applying incremental 28573 -25264> 2013-04-03 10:33:48.240175 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28573 v28573: 240 pgs: 240 active+clean; 239 MB data, 5626 MB used, 1067 GB / 1079 GB avail; 15465B/s wr, 0op/s -25263> 2013-04-03 10:33:48.240200 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28573 update_from_paxos applying incremental 28574 -25262> 2013-04-03 10:33:48.240211 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28574 v28574: 240 pgs: 240 active+clean; 239 MB data, 5627 MB used, 1067 GB / 1079 GB avail; 1941B/s wr, 0op/s -25261> 2013-04-03 10:33:48.240237 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28574 update_from_paxos applying incremental 28575 -25260> 2013-04-03 10:33:48.240248 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28575 v28575: 240 pgs: 240 active+clean; 239 MB data, 5627 MB used, 1067 GB / 1079 GB avail; 16685B/s wr, 2op/s -25259> 2013-04-03 10:33:48.240274 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28575 update_from_paxos applying incremental 28576 -25258> 2013-04-03 10:33:48.240282 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28576 v28576: 240 pgs: 240 active+clean; 239 MB data, 5627 MB used, 1067 GB / 1079 GB avail; 14353B/s wr, 2op/s -25257> 2013-04-03 10:33:48.240317 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28576 update_from_paxos applying incremental 28577 -25256> 2013-04-03 10:33:48.240326 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28577 v28577: 240 pgs: 240 active+clean; 239 MB data, 5627 MB used, 1067 GB / 1079 GB avail; 31418B/s wr, 0op/s -25255> 2013-04-03 10:33:48.240351 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28577 update_from_paxos applying incremental 28578 -25254> 2013-04-03 10:33:48.240359 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28578 v28578: 240 pgs: 240 active+clean; 239 MB data, 5627 MB used, 1067 GB / 1079 GB avail; 46754B/s wr, 1op/s -25253> 2013-04-03 10:33:48.240384 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28578 update_from_paxos applying incremental 28579 -25252> 2013-04-03 10:33:48.240392 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28579 v28579: 240 pgs: 240 active+clean; 239 MB data, 5627 MB used, 1067 GB / 1079 GB avail; 1582B/s wr, 0op/s -25251> 2013-04-03 10:33:48.240418 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28579 update_from_paxos applying incremental 28580 -25250> 2013-04-03 10:33:48.240426 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28580 v28580: 240 pgs: 240 active+clean; 239 MB data, 5627 MB used, 1067 GB / 1079 GB avail; 511B/s wr, 0op/s -25249> 2013-04-03 10:33:48.240451 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28580 update_from_paxos applying incremental 28581 -25248> 2013-04-03 10:33:48.240459 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28581 v28581: 240 pgs: 240 active+clean; 239 MB data, 5627 MB used, 1067 GB / 1079 GB avail; 279B/s wr, 0op/s -25247> 2013-04-03 10:33:48.240485 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28581 update_from_paxos applying incremental 28582 -25246> 2013-04-03 10:33:48.240493 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28582 v28582: 240 pgs: 240 active+clean; 239 MB data, 5627 MB used, 1067 GB / 1079 GB avail; 25520B/s wr, 0op/s -25245> 2013-04-03 10:33:48.240519 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28582 update_from_paxos applying incremental 28583 -25244> 2013-04-03 10:33:48.240527 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28583 v28583: 240 pgs: 240 active+clean; 239 MB data, 5627 MB used, 1067 GB / 1079 GB avail; 35100B/s wr, 0op/s -25243> 2013-04-03 10:33:48.240553 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28583 update_from_paxos applying incremental 28584 -25242> 2013-04-03 10:33:48.240562 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28584 v28584: 240 pgs: 240 active+clean; 239 MB data, 5627 MB used, 1067 GB / 1079 GB avail; 10405B/s wr, 0op/s -25241> 2013-04-03 10:33:48.240595 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28584 update_from_paxos applying incremental 28585 -25240> 2013-04-03 10:33:48.240607 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28585 v28585: 240 pgs: 240 active+clean; 239 MB data, 5627 MB used, 1067 GB / 1079 GB avail; 464B/s wr, 0op/s -25239> 2013-04-03 10:33:48.240632 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28585 update_from_paxos applying incremental 28586 -25238> 2013-04-03 10:33:48.240640 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28586 v28586: 240 pgs: 240 active+clean; 239 MB data, 5628 MB used, 1067 GB / 1079 GB avail -25237> 2013-04-03 10:33:48.240665 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28586 update_from_paxos applying incremental 28587 -25236> 2013-04-03 10:33:48.240673 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28587 v28587: 240 pgs: 240 active+clean; 239 MB data, 5628 MB used, 1067 GB / 1079 GB avail -25235> 2013-04-03 10:33:48.240698 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28587 update_from_paxos applying incremental 28588 -25234> 2013-04-03 10:33:48.240706 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28588 v28588: 240 pgs: 240 active+clean; 239 MB data, 5628 MB used, 1067 GB / 1079 GB avail -25233> 2013-04-03 10:33:48.240731 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28588 update_from_paxos applying incremental 28589 -25232> 2013-04-03 10:33:48.240739 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28589 v28589: 240 pgs: 240 active+clean; 240 MB data, 5628 MB used, 1067 GB / 1079 GB avail; 55101B/s wr, 0op/s -25231> 2013-04-03 10:33:48.240765 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28589 update_from_paxos applying incremental 28590 -25230> 2013-04-03 10:33:48.240773 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28590 v28590: 240 pgs: 240 active+clean; 240 MB data, 5628 MB used, 1067 GB / 1079 GB avail; 43002B/s wr, 0op/s -25229> 2013-04-03 10:33:48.240798 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28590 update_from_paxos applying incremental 28591 -25228> 2013-04-03 10:33:48.240804 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28591 v28591: 240 pgs: 240 active+clean; 240 MB data, 5628 MB used, 1067 GB / 1079 GB avail; 477B/s wr, 0op/s -25227> 2013-04-03 10:33:48.240829 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28591 update_from_paxos applying incremental 28592 -25226> 2013-04-03 10:33:48.240837 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28592 v28592: 240 pgs: 240 active+clean; 240 MB data, 5628 MB used, 1067 GB / 1079 GB avail; 327B/s wr, 0op/s -25225> 2013-04-03 10:33:48.240869 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28592 update_from_paxos applying incremental 28593 -25224> 2013-04-03 10:33:48.240878 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28593 v28593: 240 pgs: 240 active+clean; 240 MB data, 5629 MB used, 1067 GB / 1079 GB avail; 53389B/s wr, 0op/s -25223> 2013-04-03 10:33:48.240904 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28593 update_from_paxos applying incremental 28594 -25222> 2013-04-03 10:33:48.240914 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28594 v28594: 240 pgs: 240 active+clean; 241 MB data, 5630 MB used, 1067 GB / 1079 GB avail; 26125B/s wr, 0op/s -25221> 2013-04-03 10:33:48.240939 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28594 update_from_paxos applying incremental 28595 -25220> 2013-04-03 10:33:48.240947 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28595 v28595: 240 pgs: 240 active+clean; 241 MB data, 5630 MB used, 1067 GB / 1079 GB avail; 16496B/s wr, 0op/s -25219> 2013-04-03 10:33:48.240972 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28595 update_from_paxos applying incremental 28596 -25218> 2013-04-03 10:33:48.240980 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28596 v28596: 240 pgs: 240 active+clean; 241 MB data, 5630 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -25217> 2013-04-03 10:33:48.241005 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28596 update_from_paxos applying incremental 28597 -25216> 2013-04-03 10:33:48.241013 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28597 v28597: 240 pgs: 240 active+clean; 241 MB data, 5629 MB used, 1067 GB / 1079 GB avail; 85387B/s wr, 0op/s -25215> 2013-04-03 10:33:48.241039 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28597 update_from_paxos applying incremental 28598 -25214> 2013-04-03 10:33:48.241047 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28598 v28598: 240 pgs: 240 active+clean; 241 MB data, 5629 MB used, 1067 GB / 1079 GB avail; 49062B/s wr, 0op/s -25213> 2013-04-03 10:33:48.241072 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28598 update_from_paxos applying incremental 28599 -25212> 2013-04-03 10:33:48.241080 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28599 v28599: 240 pgs: 240 active+clean; 241 MB data, 5630 MB used, 1067 GB / 1079 GB avail; 8116B/s wr, 0op/s -25211> 2013-04-03 10:33:48.241106 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28599 update_from_paxos applying incremental 28600 -25210> 2013-04-03 10:33:48.241114 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28600 v28600: 240 pgs: 240 active+clean; 241 MB data, 5630 MB used, 1067 GB / 1079 GB avail; 7644B/s wr, 0op/s -25209> 2013-04-03 10:33:48.241149 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28600 update_from_paxos applying incremental 28601 -25208> 2013-04-03 10:33:48.241158 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28601 v28601: 240 pgs: 240 active+clean; 241 MB data, 5630 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -25207> 2013-04-03 10:33:48.241184 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28601 update_from_paxos applying incremental 28602 -25206> 2013-04-03 10:33:48.241192 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28602 v28602: 240 pgs: 240 active+clean; 241 MB data, 5630 MB used, 1067 GB / 1079 GB avail; 21079B/s wr, 0op/s -25205> 2013-04-03 10:33:48.241219 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28602 update_from_paxos applying incremental 28603 -25204> 2013-04-03 10:33:48.241228 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28603 v28603: 240 pgs: 240 active+clean; 241 MB data, 5630 MB used, 1067 GB / 1079 GB avail; 17335B/s wr, 0op/s -25203> 2013-04-03 10:33:48.241253 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28603 update_from_paxos applying incremental 28604 -25202> 2013-04-03 10:33:48.241262 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28604 v28604: 240 pgs: 240 active+clean; 241 MB data, 5630 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -25201> 2013-04-03 10:33:48.241287 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28604 update_from_paxos applying incremental 28605 -25200> 2013-04-03 10:33:48.241292 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28605 v28605: 240 pgs: 240 active+clean; 241 MB data, 5632 MB used, 1067 GB / 1079 GB avail; 81B/s wr, 0op/s -25199> 2013-04-03 10:33:48.241324 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28605 update_from_paxos applying incremental 28606 -25198> 2013-04-03 10:33:48.241333 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28606 v28606: 240 pgs: 240 active+clean; 242 MB data, 5633 MB used, 1067 GB / 1079 GB avail; 16265B/s wr, 0op/s -25197> 2013-04-03 10:33:48.241360 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28606 update_from_paxos applying incremental 28607 -25196> 2013-04-03 10:33:48.241368 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28607 v28607: 240 pgs: 240 active+clean; 242 MB data, 5633 MB used, 1067 GB / 1079 GB avail; 19113B/s wr, 0op/s -25195> 2013-04-03 10:33:48.241394 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28607 update_from_paxos applying incremental 28608 -25194> 2013-04-03 10:33:48.241402 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28608 v28608: 240 pgs: 240 active+clean; 242 MB data, 5633 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -25193> 2013-04-03 10:33:48.241427 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28608 update_from_paxos applying incremental 28609 -25192> 2013-04-03 10:33:48.241432 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28609 v28609: 240 pgs: 240 active+clean; 242 MB data, 5633 MB used, 1067 GB / 1079 GB avail; 146B/s wr, 0op/s -25191> 2013-04-03 10:33:48.241456 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28609 update_from_paxos applying incremental 28610 -25190> 2013-04-03 10:33:48.241464 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28610 v28610: 240 pgs: 240 active+clean; 242 MB data, 5633 MB used, 1067 GB / 1079 GB avail; 35828B/s wr, 0op/s -25189> 2013-04-03 10:33:48.241496 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28610 update_from_paxos applying incremental 28611 -25188> 2013-04-03 10:33:48.241504 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28611 v28611: 240 pgs: 240 active+clean; 242 MB data, 5633 MB used, 1067 GB / 1079 GB avail; 30669B/s wr, 0op/s -25187> 2013-04-03 10:33:48.241529 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28611 update_from_paxos applying incremental 28612 -25186> 2013-04-03 10:33:48.241537 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28612 v28612: 240 pgs: 240 active+clean; 243 MB data, 5633 MB used, 1067 GB / 1079 GB avail; 26428B/s wr, 0op/s -25185> 2013-04-03 10:33:48.241562 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28612 update_from_paxos applying incremental 28613 -25184> 2013-04-03 10:33:48.241571 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28613 v28613: 240 pgs: 240 active+clean; 243 MB data, 5633 MB used, 1067 GB / 1079 GB avail; 25937B/s wr, 0op/s -25183> 2013-04-03 10:33:48.241597 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28613 update_from_paxos applying incremental 28614 -25182> 2013-04-03 10:33:48.241605 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28614 v28614: 240 pgs: 240 active+clean; 243 MB data, 5633 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -25181> 2013-04-03 10:33:48.241632 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28614 update_from_paxos applying incremental 28615 -25180> 2013-04-03 10:33:48.241639 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28615 v28615: 240 pgs: 240 active+clean; 243 MB data, 5634 MB used, 1067 GB / 1079 GB avail; 14564B/s wr, 0op/s -25179> 2013-04-03 10:33:48.241663 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28615 update_from_paxos applying incremental 28616 -25178> 2013-04-03 10:33:48.241669 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28616 v28616: 240 pgs: 240 active+clean; 243 MB data, 5634 MB used, 1067 GB / 1079 GB avail; 10011B/s wr, 0op/s -25177> 2013-04-03 10:33:48.241693 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28616 update_from_paxos applying incremental 28617 -25176> 2013-04-03 10:33:48.241701 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28617 v28617: 240 pgs: 240 active+clean; 243 MB data, 5635 MB used, 1067 GB / 1079 GB avail; 14743B/s wr, 0op/s -25175> 2013-04-03 10:33:48.241727 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28617 update_from_paxos applying incremental 28618 -25174> 2013-04-03 10:33:48.241735 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28618 v28618: 240 pgs: 240 active+clean; 243 MB data, 5635 MB used, 1067 GB / 1079 GB avail; 14744B/s wr, 0op/s -25173> 2013-04-03 10:33:48.241766 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28618 update_from_paxos applying incremental 28619 -25172> 2013-04-03 10:33:48.241775 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28619 v28619: 240 pgs: 240 active+clean; 243 MB data, 5635 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -25171> 2013-04-03 10:33:48.241800 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28619 update_from_paxos applying incremental 28620 -25170> 2013-04-03 10:33:48.241808 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28620 v28620: 240 pgs: 240 active+clean; 243 MB data, 5634 MB used, 1067 GB / 1079 GB avail; 27401B/s wr, 0op/s -25169> 2013-04-03 10:33:48.241834 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28620 update_from_paxos applying incremental 28621 -25168> 2013-04-03 10:33:48.241843 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28621 v28621: 240 pgs: 240 active+clean; 243 MB data, 5634 MB used, 1067 GB / 1079 GB avail; 15995B/s wr, 0op/s -25167> 2013-04-03 10:33:48.241868 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28621 update_from_paxos applying incremental 28622 -25166> 2013-04-03 10:33:48.241877 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28622 v28622: 240 pgs: 240 active+clean; 243 MB data, 5635 MB used, 1067 GB / 1079 GB avail; 8692B/s wr, 0op/s -25165> 2013-04-03 10:33:48.241903 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28622 update_from_paxos applying incremental 28623 -25164> 2013-04-03 10:33:48.241909 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28623 v28623: 240 pgs: 240 active+clean; 243 MB data, 5637 MB used, 1067 GB / 1079 GB avail; 9501B/s wr, 0op/s -25163> 2013-04-03 10:33:48.241933 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28623 update_from_paxos applying incremental 28624 -25162> 2013-04-03 10:33:48.241942 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28624 v28624: 240 pgs: 240 active+clean; 243 MB data, 5637 MB used, 1067 GB / 1079 GB avail; 20395B/s wr, 0op/s -25161> 2013-04-03 10:33:48.241967 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28624 update_from_paxos applying incremental 28625 -25160> 2013-04-03 10:33:48.241975 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28625 v28625: 240 pgs: 240 active+clean; 243 MB data, 5637 MB used, 1067 GB / 1079 GB avail; 32845B/s wr, 0op/s -25159> 2013-04-03 10:33:48.242001 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28625 update_from_paxos applying incremental 28626 -25158> 2013-04-03 10:33:48.242009 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28626 v28626: 240 pgs: 240 active+clean; 243 MB data, 5637 MB used, 1067 GB / 1079 GB avail; 1198B/s wr, 0op/s -25157> 2013-04-03 10:33:48.242034 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28626 update_from_paxos applying incremental 28627 -25156> 2013-04-03 10:33:48.242042 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28627 v28627: 240 pgs: 240 active+clean; 243 MB data, 5637 MB used, 1067 GB / 1079 GB avail; 1206B/s wr, 0op/s -25155> 2013-04-03 10:33:48.242076 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28627 update_from_paxos applying incremental 28628 -25154> 2013-04-03 10:33:48.242086 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28628 v28628: 240 pgs: 240 active+clean; 244 MB data, 5637 MB used, 1067 GB / 1079 GB avail; 21518B/s wr, 0op/s -25153> 2013-04-03 10:33:48.242111 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28628 update_from_paxos applying incremental 28629 -25152> 2013-04-03 10:33:48.242120 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28629 v28629: 240 pgs: 240 active+clean; 244 MB data, 5637 MB used, 1067 GB / 1079 GB avail; 12265B/s wr, 0op/s -25151> 2013-04-03 10:33:48.242145 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28629 update_from_paxos applying incremental 28630 -25150> 2013-04-03 10:33:48.242157 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28630 v28630: 240 pgs: 240 active+clean; 244 MB data, 5637 MB used, 1067 GB / 1079 GB avail; 518B/s wr, 0op/s -25149> 2013-04-03 10:33:48.242183 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28630 update_from_paxos applying incremental 28631 -25148> 2013-04-03 10:33:48.242191 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28631 v28631: 240 pgs: 240 active+clean; 244 MB data, 5637 MB used, 1067 GB / 1079 GB avail; 367B/s wr, 0op/s -25147> 2013-04-03 10:33:48.242217 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28631 update_from_paxos applying incremental 28632 -25146> 2013-04-03 10:33:48.242230 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28632 v28632: 240 pgs: 240 active+clean; 243 MB data, 5637 MB used, 1067 GB / 1079 GB avail; 67312B/s wr, 1op/s -25145> 2013-04-03 10:33:48.242263 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28632 update_from_paxos applying incremental 28633 -25144> 2013-04-03 10:33:48.242278 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28633 v28633: 240 pgs: 240 active+clean; 244 MB data, 5638 MB used, 1067 GB / 1079 GB avail; 1034KB/s wr, 12op/s -25143> 2013-04-03 10:33:48.242304 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28633 update_from_paxos applying incremental 28634 -25142> 2013-04-03 10:33:48.242316 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28634 v28634: 240 pgs: 240 active+clean; 244 MB data, 5638 MB used, 1067 GB / 1079 GB avail; 616KB/s wr, 7op/s -25141> 2013-04-03 10:33:48.242343 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28634 update_from_paxos applying incremental 28635 -25140> 2013-04-03 10:33:48.242352 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28635 v28635: 240 pgs: 240 active+clean; 244 MB data, 5641 MB used, 1067 GB / 1079 GB avail; 75969B/s wr, 0op/s -25139> 2013-04-03 10:33:48.242377 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28635 update_from_paxos applying incremental 28636 -25138> 2013-04-03 10:33:48.242389 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28636 v28636: 240 pgs: 240 active+clean; 244 MB data, 5641 MB used, 1067 GB / 1079 GB avail; 123KB/s wr, 5op/s -25137> 2013-04-03 10:33:48.242417 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28636 update_from_paxos applying incremental 28637 -25136> 2013-04-03 10:33:48.242425 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28637 v28637: 240 pgs: 240 active+clean; 244 MB data, 5641 MB used, 1067 GB / 1079 GB avail; 11719B/s wr, 2op/s -25135> 2013-04-03 10:33:48.242457 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28637 update_from_paxos applying incremental 28638 -25134> 2013-04-03 10:33:48.242466 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28638 v28638: 240 pgs: 240 active+clean; 244 MB data, 5641 MB used, 1067 GB / 1079 GB avail -25133> 2013-04-03 10:33:48.242492 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28638 update_from_paxos applying incremental 28639 -25132> 2013-04-03 10:33:48.242500 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28639 v28639: 240 pgs: 240 active+clean; 244 MB data, 5640 MB used, 1067 GB / 1079 GB avail; 68150B/s wr, 0op/s -25131> 2013-04-03 10:33:48.242525 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28639 update_from_paxos applying incremental 28640 -25130> 2013-04-03 10:33:48.242533 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28640 v28640: 240 pgs: 240 active+clean; 245 MB data, 5640 MB used, 1067 GB / 1079 GB avail; 58089B/s wr, 0op/s -25129> 2013-04-03 10:33:48.242558 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28640 update_from_paxos applying incremental 28641 -25128> 2013-04-03 10:33:48.242572 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28641 v28641: 240 pgs: 240 active+clean; 245 MB data, 5640 MB used, 1067 GB / 1079 GB avail; 1210B/s wr, 0op/s -25127> 2013-04-03 10:33:48.242598 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28641 update_from_paxos applying incremental 28642 -25126> 2013-04-03 10:33:48.242606 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28642 v28642: 240 pgs: 240 active+clean; 245 MB data, 5640 MB used, 1067 GB / 1079 GB avail; 1184B/s wr, 0op/s -25125> 2013-04-03 10:33:48.242631 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28642 update_from_paxos applying incremental 28643 -25124> 2013-04-03 10:33:48.242640 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28643 v28643: 240 pgs: 240 active+clean; 245 MB data, 5640 MB used, 1067 GB / 1079 GB avail; 11407B/s wr, 0op/s -25123> 2013-04-03 10:33:48.242665 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28643 update_from_paxos applying incremental 28644 -25122> 2013-04-03 10:33:48.242673 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28644 v28644: 240 pgs: 240 active+clean; 245 MB data, 5640 MB used, 1067 GB / 1079 GB avail; 8191B/s wr, 0op/s -25121> 2013-04-03 10:33:48.242709 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28644 update_from_paxos applying incremental 28645 -25120> 2013-04-03 10:33:48.242719 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28645 v28645: 240 pgs: 240 active+clean; 245 MB data, 5640 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -25119> 2013-04-03 10:33:48.242745 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28645 update_from_paxos applying incremental 28646 -25118> 2013-04-03 10:33:48.242754 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28646 v28646: 240 pgs: 240 active+clean; 246 MB data, 5640 MB used, 1067 GB / 1079 GB avail; 106KB/s wr, 0op/s -25117> 2013-04-03 10:33:48.242778 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28646 update_from_paxos applying incremental 28647 -25116> 2013-04-03 10:33:48.242786 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28647 v28647: 240 pgs: 240 active+clean; 246 MB data, 5640 MB used, 1067 GB / 1079 GB avail; 73625B/s wr, 0op/s -25115> 2013-04-03 10:33:48.242812 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28647 update_from_paxos applying incremental 28648 -25114> 2013-04-03 10:33:48.242819 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28648 v28648: 240 pgs: 240 active+clean; 246 MB data, 5641 MB used, 1067 GB / 1079 GB avail; 12286B/s wr, 0op/s -25113> 2013-04-03 10:33:48.242844 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28648 update_from_paxos applying incremental 28649 -25112> 2013-04-03 10:33:48.242852 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28649 v28649: 240 pgs: 240 active+clean; 246 MB data, 5641 MB used, 1067 GB / 1079 GB avail; 12286B/s wr, 0op/s -25111> 2013-04-03 10:33:48.242878 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28649 update_from_paxos applying incremental 28650 -25110> 2013-04-03 10:33:48.242886 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28650 v28650: 240 pgs: 240 active+clean; 246 MB data, 5641 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -25109> 2013-04-03 10:33:48.242911 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28650 update_from_paxos applying incremental 28651 -25108> 2013-04-03 10:33:48.242920 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28651 v28651: 240 pgs: 240 active+clean; 246 MB data, 5641 MB used, 1067 GB / 1079 GB avail; 14329B/s wr, 0op/s -25107> 2013-04-03 10:33:48.242946 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28651 update_from_paxos applying incremental 28652 -25106> 2013-04-03 10:33:48.242954 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28652 v28652: 240 pgs: 240 active+clean; 246 MB data, 5641 MB used, 1067 GB / 1079 GB avail; 12279B/s wr, 0op/s -25105> 2013-04-03 10:33:48.242987 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28652 update_from_paxos applying incremental 28653 -25104> 2013-04-03 10:33:48.242995 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28653 v28653: 240 pgs: 240 active+clean; 246 MB data, 5641 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -25103> 2013-04-03 10:33:48.243021 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28653 update_from_paxos applying incremental 28654 -25102> 2013-04-03 10:33:48.243029 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28654 v28654: 240 pgs: 240 active+clean; 246 MB data, 5641 MB used, 1067 GB / 1079 GB avail; 14325B/s wr, 0op/s -25101> 2013-04-03 10:33:48.243054 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28654 update_from_paxos applying incremental 28655 -25100> 2013-04-03 10:33:48.243062 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28655 v28655: 240 pgs: 240 active+clean; 246 MB data, 5642 MB used, 1067 GB / 1079 GB avail; 11117B/s wr, 0op/s -25099> 2013-04-03 10:33:48.243087 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28655 update_from_paxos applying incremental 28656 -25098> 2013-04-03 10:33:48.243095 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28656 v28656: 240 pgs: 240 active+clean; 246 MB data, 5641 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -25097> 2013-04-03 10:33:48.243121 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28656 update_from_paxos applying incremental 28657 -25096> 2013-04-03 10:33:48.243130 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28657 v28657: 240 pgs: 240 active+clean; 246 MB data, 5642 MB used, 1067 GB / 1079 GB avail; 16635B/s wr, 0op/s -25095> 2013-04-03 10:33:48.243155 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28657 update_from_paxos applying incremental 28658 -25094> 2013-04-03 10:33:48.243164 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28658 v28658: 240 pgs: 240 active+clean; 246 MB data, 5642 MB used, 1067 GB / 1079 GB avail; 7995B/s wr, 0op/s -25093> 2013-04-03 10:33:48.243189 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28658 update_from_paxos applying incremental 28659 -25092> 2013-04-03 10:33:48.243194 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28659 v28659: 240 pgs: 240 active+clean; 246 MB data, 5641 MB used, 1067 GB / 1079 GB avail; 479B/s wr, 0op/s -25091> 2013-04-03 10:33:48.243319 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28659 update_from_paxos applying incremental 28660 -25090> 2013-04-03 10:33:48.243334 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28660 v28660: 240 pgs: 240 active+clean; 246 MB data, 5642 MB used, 1067 GB / 1079 GB avail; 18291B/s wr, 0op/s -25089> 2013-04-03 10:33:48.243363 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28660 update_from_paxos applying incremental 28661 -25088> 2013-04-03 10:33:48.243369 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28661 v28661: 240 pgs: 240 active+clean; 246 MB data, 5643 MB used, 1067 GB / 1079 GB avail; 10076B/s wr, 0op/s -25087> 2013-04-03 10:33:48.243393 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28661 update_from_paxos applying incremental 28662 -25086> 2013-04-03 10:33:48.243401 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28662 v28662: 240 pgs: 240 active+clean; 246 MB data, 5644 MB used, 1067 GB / 1079 GB avail; 3673B/s wr, 0op/s -25085> 2013-04-03 10:33:48.243434 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28662 update_from_paxos applying incremental 28663 -25084> 2013-04-03 10:33:48.243442 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28663 v28663: 240 pgs: 240 active+clean; 247 MB data, 5644 MB used, 1067 GB / 1079 GB avail; 6068B/s wr, 0op/s -25083> 2013-04-03 10:33:48.243468 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28663 update_from_paxos applying incremental 28664 -25082> 2013-04-03 10:33:48.243476 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28664 v28664: 240 pgs: 240 active+clean; 247 MB data, 5644 MB used, 1067 GB / 1079 GB avail; 4997B/s wr, 0op/s -25081> 2013-04-03 10:33:48.243501 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28664 update_from_paxos applying incremental 28665 -25080> 2013-04-03 10:33:48.243509 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28665 v28665: 240 pgs: 240 active+clean; 247 MB data, 5644 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -25079> 2013-04-03 10:33:48.243534 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28665 update_from_paxos applying incremental 28666 -25078> 2013-04-03 10:33:48.243542 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28666 v28666: 240 pgs: 240 active+clean; 247 MB data, 5644 MB used, 1067 GB / 1079 GB avail; 3527B/s wr, 0op/s -25077> 2013-04-03 10:33:48.243568 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28666 update_from_paxos applying incremental 28667 -25076> 2013-04-03 10:33:48.243576 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28667 v28667: 240 pgs: 240 active+clean; 247 MB data, 5644 MB used, 1067 GB / 1079 GB avail; 3106B/s wr, 0op/s -25075> 2013-04-03 10:33:48.243601 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28667 update_from_paxos applying incremental 28668 -25074> 2013-04-03 10:33:48.243609 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28668 v28668: 240 pgs: 240 active+clean; 247 MB data, 5644 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -25073> 2013-04-03 10:33:48.243635 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28668 update_from_paxos applying incremental 28669 -25072> 2013-04-03 10:33:48.243640 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28669 v28669: 240 pgs: 240 active+clean; 247 MB data, 5644 MB used, 1067 GB / 1079 GB avail; 85B/s wr, 0op/s -25071> 2013-04-03 10:33:48.243665 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28669 update_from_paxos applying incremental 28670 -25070> 2013-04-03 10:33:48.243674 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28670 v28670: 240 pgs: 240 active+clean; 247 MB data, 5644 MB used, 1067 GB / 1079 GB avail; 1389B/s wr, 0op/s -25069> 2013-04-03 10:33:48.243699 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28670 update_from_paxos applying incremental 28671 -25068> 2013-04-03 10:33:48.243707 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28671 v28671: 240 pgs: 240 active+clean; 247 MB data, 5644 MB used, 1067 GB / 1079 GB avail; 2563B/s wr, 0op/s -25067> 2013-04-03 10:33:48.243742 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28671 update_from_paxos applying incremental 28672 -25066> 2013-04-03 10:33:48.243751 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28672 v28672: 240 pgs: 240 active+clean; 247 MB data, 5644 MB used, 1067 GB / 1079 GB avail; 623B/s wr, 0op/s -25065> 2013-04-03 10:33:48.243777 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28672 update_from_paxos applying incremental 28673 -25064> 2013-04-03 10:33:48.243786 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28673 v28673: 240 pgs: 240 active+clean; 247 MB data, 5644 MB used, 1067 GB / 1079 GB avail; 2108B/s wr, 0op/s -25063> 2013-04-03 10:33:48.243811 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28673 update_from_paxos applying incremental 28674 -25062> 2013-04-03 10:33:48.243816 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28674 v28674: 240 pgs: 240 active+clean; 247 MB data, 5644 MB used, 1067 GB / 1079 GB avail; 1581B/s wr, 0op/s -25061> 2013-04-03 10:33:48.243843 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28674 update_from_paxos applying incremental 28675 -25060> 2013-04-03 10:33:48.243852 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28675 v28675: 240 pgs: 240 active+clean; 247 MB data, 5644 MB used, 1067 GB / 1079 GB avail; 727B/s wr, 0op/s -25059> 2013-04-03 10:33:48.243878 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28675 update_from_paxos applying incremental 28676 -25058> 2013-04-03 10:33:48.243886 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28676 v28676: 240 pgs: 240 active+clean; 247 MB data, 5645 MB used, 1067 GB / 1079 GB avail; 1859B/s wr, 0op/s -25057> 2013-04-03 10:33:48.243912 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28676 update_from_paxos applying incremental 28677 -25056> 2013-04-03 10:33:48.243920 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28677 v28677: 240 pgs: 240 active+clean; 247 MB data, 5645 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -25055> 2013-04-03 10:33:48.243945 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28677 update_from_paxos applying incremental 28678 -25054> 2013-04-03 10:33:48.243953 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28678 v28678: 240 pgs: 240 active+clean; 247 MB data, 5645 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -25053> 2013-04-03 10:33:48.243978 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28678 update_from_paxos applying incremental 28679 -25052> 2013-04-03 10:33:48.243988 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28679 v28679: 240 pgs: 240 active+clean; 247 MB data, 5645 MB used, 1067 GB / 1079 GB avail; 1564B/s wr, 0op/s -25051> 2013-04-03 10:33:48.244015 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28679 update_from_paxos applying incremental 28680 -25050> 2013-04-03 10:33:48.244023 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28680 v28680: 240 pgs: 240 active+clean; 247 MB data, 5645 MB used, 1067 GB / 1079 GB avail; 1291B/s wr, 0op/s -25049> 2013-04-03 10:33:48.244055 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28680 update_from_paxos applying incremental 28681 -25048> 2013-04-03 10:33:48.244065 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28681 v28681: 240 pgs: 240 active+clean; 247 MB data, 5645 MB used, 1067 GB / 1079 GB avail; 370B/s wr, 0op/s -25047> 2013-04-03 10:33:48.244091 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28681 update_from_paxos applying incremental 28682 -25046> 2013-04-03 10:33:48.244101 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28682 v28682: 240 pgs: 240 active+clean; 247 MB data, 5645 MB used, 1067 GB / 1079 GB avail; 42766B/s wr, 0op/s -25045> 2013-04-03 10:33:48.244126 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28682 update_from_paxos applying incremental 28683 -25044> 2013-04-03 10:33:48.244135 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28683 v28683: 240 pgs: 240 active+clean; 247 MB data, 5645 MB used, 1067 GB / 1079 GB avail; 151KB/s wr, 4op/s -25043> 2013-04-03 10:33:48.244160 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28683 update_from_paxos applying incremental 28684 -25042> 2013-04-03 10:33:48.244170 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28684 v28684: 240 pgs: 240 active+clean; 250 MB data, 5646 MB used, 1067 GB / 1079 GB avail; 884KB/s wr, 4op/s -25041> 2013-04-03 10:33:48.244195 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28684 update_from_paxos applying incremental 28685 -25040> 2013-04-03 10:33:48.244204 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28685 v28685: 240 pgs: 240 active+clean; 250 MB data, 5646 MB used, 1067 GB / 1079 GB avail; 551KB/s wr, 1op/s -25039> 2013-04-03 10:33:48.244229 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28685 update_from_paxos applying incremental 28686 -25038> 2013-04-03 10:33:48.244237 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28686 v28686: 240 pgs: 240 active+clean; 250 MB data, 5646 MB used, 1067 GB / 1079 GB avail; 3063B/s wr, 0op/s -25037> 2013-04-03 10:33:48.244268 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28686 update_from_paxos applying incremental 28687 -25036> 2013-04-03 10:33:48.244276 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28687 v28687: 240 pgs: 240 active+clean; 250 MB data, 5646 MB used, 1067 GB / 1079 GB avail; 1925B/s wr, 0op/s -25035> 2013-04-03 10:33:48.244302 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28687 update_from_paxos applying incremental 28688 -25034> 2013-04-03 10:33:48.244310 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28688 v28688: 240 pgs: 240 active+clean; 250 MB data, 5646 MB used, 1067 GB / 1079 GB avail; 3081B/s wr, 0op/s -25033> 2013-04-03 10:33:48.244335 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28688 update_from_paxos applying incremental 28689 -25032> 2013-04-03 10:33:48.244345 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28689 v28689: 240 pgs: 240 active+clean; 250 MB data, 5646 MB used, 1067 GB / 1079 GB avail; 1463B/s wr, 0op/s -25031> 2013-04-03 10:33:48.244371 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28689 update_from_paxos applying incremental 28690 -25030> 2013-04-03 10:33:48.244380 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28690 v28690: 240 pgs: 240 active+clean; 250 MB data, 5646 MB used, 1067 GB / 1079 GB avail; 1176B/s wr, 0op/s -25029> 2013-04-03 10:33:48.244405 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28690 update_from_paxos applying incremental 28691 -25028> 2013-04-03 10:33:48.244414 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28691 v28691: 240 pgs: 240 active+clean; 250 MB data, 5646 MB used, 1067 GB / 1079 GB avail; 1516B/s wr, 0op/s -25027> 2013-04-03 10:33:48.244439 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28691 update_from_paxos applying incremental 28692 -25026> 2013-04-03 10:33:48.244447 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28692 v28692: 240 pgs: 240 active+clean; 250 MB data, 5646 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -25025> 2013-04-03 10:33:48.244475 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28692 update_from_paxos applying incremental 28693 -25024> 2013-04-03 10:33:48.244484 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28693 v28693: 240 pgs: 240 active+clean; 250 MB data, 5646 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -25023> 2013-04-03 10:33:48.244510 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28693 update_from_paxos applying incremental 28694 -25022> 2013-04-03 10:33:48.244518 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28694 v28694: 240 pgs: 240 active+clean; 250 MB data, 5646 MB used, 1067 GB / 1079 GB avail; 85323B/s wr, 0op/s -25021> 2013-04-03 10:33:48.244553 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28694 update_from_paxos applying incremental 28695 -25020> 2013-04-03 10:33:48.244562 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28695 v28695: 240 pgs: 240 active+clean; 250 MB data, 5646 MB used, 1067 GB / 1079 GB avail; 57458B/s wr, 0op/s -25019> 2013-04-03 10:33:48.244588 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28695 update_from_paxos applying incremental 28696 -25018> 2013-04-03 10:33:48.244596 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28696 v28696: 240 pgs: 240 active+clean; 250 MB data, 5647 MB used, 1067 GB / 1079 GB avail; 465B/s wr, 0op/s -25017> 2013-04-03 10:33:48.244621 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28696 update_from_paxos applying incremental 28697 -25016> 2013-04-03 10:33:48.244629 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28697 v28697: 240 pgs: 240 active+clean; 250 MB data, 5647 MB used, 1067 GB / 1079 GB avail; 1925B/s wr, 0op/s -25015> 2013-04-03 10:33:48.244654 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28697 update_from_paxos applying incremental 28698 -25014> 2013-04-03 10:33:48.244662 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28698 v28698: 240 pgs: 240 active+clean; 250 MB data, 5647 MB used, 1067 GB / 1079 GB avail; 1686B/s wr, 0op/s -25013> 2013-04-03 10:33:48.244687 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28698 update_from_paxos applying incremental 28699 -25012> 2013-04-03 10:33:48.244696 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28699 v28699: 240 pgs: 240 active+clean; 251 MB data, 5647 MB used, 1067 GB / 1079 GB avail; 630B/s wr, 0op/s -25011> 2013-04-03 10:33:48.244721 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28699 update_from_paxos applying incremental 28700 -25010> 2013-04-03 10:33:48.244729 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28700 v28700: 240 pgs: 240 active+clean; 251 MB data, 5647 MB used, 1067 GB / 1079 GB avail; 967B/s wr, 0op/s -25009> 2013-04-03 10:33:48.244755 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28700 update_from_paxos applying incremental 28701 -25008> 2013-04-03 10:33:48.244763 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28701 v28701: 240 pgs: 240 active+clean; 251 MB data, 5647 MB used, 1067 GB / 1079 GB avail; 1564B/s wr, 0op/s -25007> 2013-04-03 10:33:48.244788 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28701 update_from_paxos applying incremental 28702 -25006> 2013-04-03 10:33:48.244794 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28702 v28702: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 1206B/s wr, 0op/s -25005> 2013-04-03 10:33:48.244817 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28702 update_from_paxos applying incremental 28703 -25004> 2013-04-03 10:33:48.244825 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28703 v28703: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 630B/s wr, 0op/s -25003> 2013-04-03 10:33:48.244850 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28703 update_from_paxos applying incremental 28704 -25002> 2013-04-03 10:33:48.244858 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28704 v28704: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 2723B/s wr, 0op/s -25001> 2013-04-03 10:33:48.244892 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28704 update_from_paxos applying incremental 28705 -25000> 2013-04-03 10:33:48.244901 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28705 v28705: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -24999> 2013-04-03 10:33:48.244927 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28705 update_from_paxos applying incremental 28706 -24998> 2013-04-03 10:33:48.244935 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28706 v28706: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -24997> 2013-04-03 10:33:48.244960 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28706 update_from_paxos applying incremental 28707 -24996> 2013-04-03 10:33:48.244968 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28707 v28707: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 1171B/s wr, 0op/s -24995> 2013-04-03 10:33:48.244993 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28707 update_from_paxos applying incremental 28708 -24994> 2013-04-03 10:33:48.245004 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28708 v28708: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 1982B/s wr, 0op/s -24993> 2013-04-03 10:33:48.245031 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28708 update_from_paxos applying incremental 28709 -24992> 2013-04-03 10:33:48.245039 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28709 v28709: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 1982B/s wr, 0op/s -24991> 2013-04-03 10:33:48.245065 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28709 update_from_paxos applying incremental 28710 -24990> 2013-04-03 10:33:48.245073 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28710 v28710: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -24989> 2013-04-03 10:33:48.245101 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28710 update_from_paxos applying incremental 28711 -24988> 2013-04-03 10:33:48.245110 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28711 v28711: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 1632B/s wr, 0op/s -24987> 2013-04-03 10:33:48.245136 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28711 update_from_paxos applying incremental 28712 -24986> 2013-04-03 10:33:48.245144 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28712 v28712: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 1261B/s wr, 0op/s -24985> 2013-04-03 10:33:48.245176 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28712 update_from_paxos applying incremental 28713 -24984> 2013-04-03 10:33:48.245185 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28713 v28713: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 761B/s wr, 0op/s -24983> 2013-04-03 10:33:48.245210 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28713 update_from_paxos applying incremental 28714 -24982> 2013-04-03 10:33:48.245219 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28714 v28714: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 3047B/s wr, 0op/s -24981> 2013-04-03 10:33:48.245244 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28714 update_from_paxos applying incremental 28715 -24980> 2013-04-03 10:33:48.245252 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28715 v28715: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 2130B/s wr, 0op/s -24979> 2013-04-03 10:33:48.245277 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28715 update_from_paxos applying incremental 28716 -24978> 2013-04-03 10:33:48.245285 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28716 v28716: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 896B/s wr, 0op/s -24977> 2013-04-03 10:33:48.245310 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28716 update_from_paxos applying incremental 28717 -24976> 2013-04-03 10:33:48.245328 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28717 v28717: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -24975> 2013-04-03 10:33:48.245357 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28717 update_from_paxos applying incremental 28718 -24974> 2013-04-03 10:33:48.245365 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28718 v28718: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 577B/s wr, 0op/s -24973> 2013-04-03 10:33:48.245391 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28718 update_from_paxos applying incremental 28719 -24972> 2013-04-03 10:33:48.245399 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28719 v28719: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 16903B/s wr, 0op/s -24971> 2013-04-03 10:33:48.245425 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28719 update_from_paxos applying incremental 28720 -24970> 2013-04-03 10:33:48.245433 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28720 v28720: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 10195B/s wr, 0op/s -24969> 2013-04-03 10:33:48.245471 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28720 update_from_paxos applying incremental 28721 -24968> 2013-04-03 10:33:48.245480 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28721 v28721: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 1162B/s wr, 0op/s -24967> 2013-04-03 10:33:48.245506 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28721 update_from_paxos applying incremental 28722 -24966> 2013-04-03 10:33:48.245514 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28722 v28722: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -24965> 2013-04-03 10:33:48.245539 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28722 update_from_paxos applying incremental 28723 -24964> 2013-04-03 10:33:48.245547 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28723 v28723: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -24963> 2013-04-03 10:33:48.245572 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28723 update_from_paxos applying incremental 28724 -24962> 2013-04-03 10:33:48.245580 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28724 v28724: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 5815B/s wr, 0op/s -24961> 2013-04-03 10:33:48.245605 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28724 update_from_paxos applying incremental 28725 -24960> 2013-04-03 10:33:48.245613 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28725 v28725: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 4127B/s wr, 0op/s -24959> 2013-04-03 10:33:48.245638 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28725 update_from_paxos applying incremental 28726 -24958> 2013-04-03 10:33:48.245646 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28726 v28726: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 4661B/s wr, 0op/s -24957> 2013-04-03 10:33:48.245672 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28726 update_from_paxos applying incremental 28727 -24956> 2013-04-03 10:33:48.245680 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28727 v28727: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 4808B/s wr, 0op/s -24955> 2013-04-03 10:33:48.245705 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28727 update_from_paxos applying incremental 28728 -24954> 2013-04-03 10:33:48.245713 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28728 v28728: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 630B/s wr, 0op/s -24953> 2013-04-03 10:33:48.245742 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28728 update_from_paxos applying incremental 28729 -24952> 2013-04-03 10:33:48.245764 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28729 v28729: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 970B/s wr, 9op/s -24951> 2013-04-03 10:33:48.245798 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28729 update_from_paxos applying incremental 28730 -24950> 2013-04-03 10:33:48.245813 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28730 v28730: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 306B/s wr, 25op/s -24949> 2013-04-03 10:33:48.245840 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28730 update_from_paxos applying incremental 28731 -24948> 2013-04-03 10:33:48.245857 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28731 v28731: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 0B/s wr, 34op/s -24947> 2013-04-03 10:33:48.245883 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28731 update_from_paxos applying incremental 28732 -24946> 2013-04-03 10:33:48.245891 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28732 v28732: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 1584B/s wr, 8op/s -24945> 2013-04-03 10:33:48.245923 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28732 update_from_paxos applying incremental 28733 -24944> 2013-04-03 10:33:48.245931 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28733 v28733: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 1181B/s wr, 0op/s -24943> 2013-04-03 10:33:48.245957 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28733 update_from_paxos applying incremental 28734 -24942> 2013-04-03 10:33:48.245965 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28734 v28734: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -24941> 2013-04-03 10:33:48.245990 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28734 update_from_paxos applying incremental 28735 -24940> 2013-04-03 10:33:48.245998 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28735 v28735: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -24939> 2013-04-03 10:33:48.246023 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28735 update_from_paxos applying incremental 28736 -24938> 2013-04-03 10:33:48.246034 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28736 v28736: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -24937> 2013-04-03 10:33:48.246060 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28736 update_from_paxos applying incremental 28737 -24936> 2013-04-03 10:33:48.246068 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28737 v28737: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 2045B/s wr, 0op/s -24935> 2013-04-03 10:33:48.246094 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28737 update_from_paxos applying incremental 28738 -24934> 2013-04-03 10:33:48.246102 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28738 v28738: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1828B/s wr, 0op/s -24933> 2013-04-03 10:33:48.246127 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28738 update_from_paxos applying incremental 28739 -24932> 2013-04-03 10:33:48.246135 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28739 v28739: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s -24931> 2013-04-03 10:33:48.246161 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28739 update_from_paxos applying incremental 28740 -24930> 2013-04-03 10:33:48.246169 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28740 v28740: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 3276B/s wr, 0op/s -24929> 2013-04-03 10:33:48.246203 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28740 update_from_paxos applying incremental 28741 -24928> 2013-04-03 10:33:48.246212 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28741 v28741: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 3274B/s wr, 0op/s -24927> 2013-04-03 10:33:48.246238 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28741 update_from_paxos applying incremental 28742 -24926> 2013-04-03 10:33:48.246246 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28742 v28742: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 1239B/s wr, 0op/s -24925> 2013-04-03 10:33:48.246271 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28742 update_from_paxos applying incremental 28743 -24924> 2013-04-03 10:33:48.246279 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28743 v28743: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 1159B/s wr, 0op/s -24923> 2013-04-03 10:33:48.246304 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28743 update_from_paxos applying incremental 28744 -24922> 2013-04-03 10:33:48.246313 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28744 v28744: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1395B/s wr, 0op/s -24921> 2013-04-03 10:33:48.246338 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28744 update_from_paxos applying incremental 28745 -24920> 2013-04-03 10:33:48.246346 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28745 v28745: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -24919> 2013-04-03 10:33:48.246372 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28745 update_from_paxos applying incremental 28746 -24918> 2013-04-03 10:33:48.246381 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28746 v28746: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -24917> 2013-04-03 10:33:48.246408 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28746 update_from_paxos applying incremental 28747 -24916> 2013-04-03 10:33:48.246417 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28747 v28747: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1443B/s wr, 0op/s -24915> 2013-04-03 10:33:48.246443 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28747 update_from_paxos applying incremental 28748 -24914> 2013-04-03 10:33:48.246451 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28748 v28748: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1159B/s wr, 0op/s -24913> 2013-04-03 10:33:48.246484 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28748 update_from_paxos applying incremental 28749 -24912> 2013-04-03 10:33:48.246493 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28749 v28749: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 1242B/s wr, 0op/s -24911> 2013-04-03 10:33:48.246519 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28749 update_from_paxos applying incremental 28750 -24910> 2013-04-03 10:33:48.246527 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28750 v28750: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 1156B/s wr, 0op/s -24909> 2013-04-03 10:33:48.246553 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28750 update_from_paxos applying incremental 28751 -24908> 2013-04-03 10:33:48.246561 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28751 v28751: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 1505B/s wr, 0op/s -24907> 2013-04-03 10:33:48.246586 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28751 update_from_paxos applying incremental 28752 -24906> 2013-04-03 10:33:48.246594 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28752 v28752: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 1515B/s wr, 0op/s -24905> 2013-04-03 10:33:48.246620 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28752 update_from_paxos applying incremental 28753 -24904> 2013-04-03 10:33:48.246628 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28753 v28753: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 1719B/s wr, 0op/s -24903> 2013-04-03 10:33:48.246653 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28753 update_from_paxos applying incremental 28754 -24902> 2013-04-03 10:33:48.246661 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28754 v28754: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1190B/s wr, 0op/s -24901> 2013-04-03 10:33:48.246687 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28754 update_from_paxos applying incremental 28755 -24900> 2013-04-03 10:33:48.246696 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28755 v28755: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1177B/s wr, 0op/s -24899> 2013-04-03 10:33:48.246722 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28755 update_from_paxos applying incremental 28756 -24898> 2013-04-03 10:33:48.246730 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28756 v28756: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1437B/s wr, 0op/s -24897> 2013-04-03 10:33:48.246762 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28756 update_from_paxos applying incremental 28757 -24896> 2013-04-03 10:33:48.246771 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28757 v28757: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -24895> 2013-04-03 10:33:48.246796 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28757 update_from_paxos applying incremental 28758 -24894> 2013-04-03 10:33:48.246809 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28758 v28758: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 20595B/s wr, 0op/s -24893> 2013-04-03 10:33:48.246835 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28758 update_from_paxos applying incremental 28759 -24892> 2013-04-03 10:33:48.246843 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28759 v28759: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 44845B/s wr, 1op/s -24891> 2013-04-03 10:33:48.246868 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28759 update_from_paxos applying incremental 28760 -24890> 2013-04-03 10:33:48.246873 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28760 v28760: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 9596B/s wr, 1op/s -24889> 2013-04-03 10:33:48.246897 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28760 update_from_paxos applying incremental 28761 -24888> 2013-04-03 10:33:48.246908 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28761 v28761: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail -24887> 2013-04-03 10:33:48.246933 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28761 update_from_paxos applying incremental 28762 -24886> 2013-04-03 10:33:48.246941 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28762 v28762: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 7841B/s wr, 0op/s -24885> 2013-04-03 10:33:48.246967 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28762 update_from_paxos applying incremental 28763 -24884> 2013-04-03 10:33:48.246975 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28763 v28763: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 14210B/s wr, 2op/s -24883> 2013-04-03 10:33:48.247011 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28763 update_from_paxos applying incremental 28764 -24882> 2013-04-03 10:33:48.247017 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28764 v28764: 240 pgs: 240 active+clean; 251 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 4182B/s wr, 1op/s -24881> 2013-04-03 10:33:48.247043 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28764 update_from_paxos applying incremental 28765 -24880> 2013-04-03 10:33:48.247052 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28765 v28765: 240 pgs: 240 active+clean; 251 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 877B/s wr, 0op/s -24879> 2013-04-03 10:33:48.247078 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28765 update_from_paxos applying incremental 28766 -24878> 2013-04-03 10:33:48.247086 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28766 v28766: 240 pgs: 240 active+clean; 251 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2720B/s wr, 0op/s -24877> 2013-04-03 10:33:48.247364 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28766 update_from_paxos applying incremental 28767 -24876> 2013-04-03 10:33:48.247378 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28767 v28767: 240 pgs: 240 active+clean; 251 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2893B/s wr, 0op/s -24875> 2013-04-03 10:33:48.247406 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28767 update_from_paxos applying incremental 28768 -24874> 2013-04-03 10:33:48.247415 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28768 v28768: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 1182B/s wr, 0op/s -24873> 2013-04-03 10:33:48.247441 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28768 update_from_paxos applying incremental 28769 -24872> 2013-04-03 10:33:48.247449 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28769 v28769: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 1578B/s wr, 0op/s -24871> 2013-04-03 10:33:48.247474 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28769 update_from_paxos applying incremental 28770 -24870> 2013-04-03 10:33:48.247483 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28770 v28770: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 3735B/s wr, 0op/s -24869> 2013-04-03 10:33:48.247508 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28770 update_from_paxos applying incremental 28771 -24868> 2013-04-03 10:33:48.247516 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28771 v28771: 240 pgs: 240 active+clean; 251 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 1344B/s wr, 0op/s -24867> 2013-04-03 10:33:48.247542 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28771 update_from_paxos applying incremental 28772 -24866> 2013-04-03 10:33:48.247552 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28772 v28772: 240 pgs: 240 active+clean; 251 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 562B/s wr, 0op/s -24865> 2013-04-03 10:33:48.247586 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28772 update_from_paxos applying incremental 28773 -24864> 2013-04-03 10:33:48.247594 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28773 v28773: 240 pgs: 240 active+clean; 252 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 39604B/s wr, 0op/s -24863> 2013-04-03 10:33:48.247620 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28773 update_from_paxos applying incremental 28774 -24862> 2013-04-03 10:33:48.247630 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28774 v28774: 240 pgs: 240 active+clean; 252 MB data, 5651 MB used, 1067 GB / 1079 GB avail; 32488B/s wr, 0op/s -24861> 2013-04-03 10:33:48.247655 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28774 update_from_paxos applying incremental 28775 -24860> 2013-04-03 10:33:48.247663 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28775 v28775: 240 pgs: 240 active+clean; 252 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -24859> 2013-04-03 10:33:48.247689 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28775 update_from_paxos applying incremental 28776 -24858> 2013-04-03 10:33:48.247697 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28776 v28776: 240 pgs: 240 active+clean; 252 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 1288B/s wr, 0op/s -24857> 2013-04-03 10:33:48.247722 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28776 update_from_paxos applying incremental 28777 -24856> 2013-04-03 10:33:48.247733 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28777 v28777: 240 pgs: 240 active+clean; 252 MB data, 5652 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -24855> 2013-04-03 10:33:48.247759 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28777 update_from_paxos applying incremental 28778 -24854> 2013-04-03 10:33:48.247769 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28778 v28778: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 420B/s wr, 1op/s -24853> 2013-04-03 10:33:48.247795 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28778 update_from_paxos applying incremental 28779 -24852> 2013-04-03 10:33:48.247803 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28779 v28779: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail -24851> 2013-04-03 10:33:48.247835 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28779 update_from_paxos applying incremental 28780 -24850> 2013-04-03 10:33:48.247848 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28780 v28780: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1829B/s wr, 0op/s -24849> 2013-04-03 10:33:48.247874 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28780 update_from_paxos applying incremental 28781 -24848> 2013-04-03 10:33:48.247882 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28781 v28781: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1792B/s wr, 0op/s -24847> 2013-04-03 10:33:48.247907 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28781 update_from_paxos applying incremental 28782 -24846> 2013-04-03 10:33:48.247915 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28782 v28782: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 3553B/s wr, 0op/s -24845> 2013-04-03 10:33:48.247942 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28782 update_from_paxos applying incremental 28783 -24844> 2013-04-03 10:33:48.247950 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28783 v28783: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 2129B/s wr, 0op/s -24843> 2013-04-03 10:33:48.247975 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28783 update_from_paxos applying incremental 28784 -24842> 2013-04-03 10:33:48.247984 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28784 v28784: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 515B/s wr, 0op/s -24841> 2013-04-03 10:33:48.248010 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28784 update_from_paxos applying incremental 28785 -24840> 2013-04-03 10:33:48.248018 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28785 v28785: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1443B/s wr, 0op/s -24839> 2013-04-03 10:33:48.248044 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28785 update_from_paxos applying incremental 28786 -24838> 2013-04-03 10:33:48.248052 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28786 v28786: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1290B/s wr, 0op/s -24837> 2013-04-03 10:33:48.248088 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28786 update_from_paxos applying incremental 28787 -24836> 2013-04-03 10:33:48.248097 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28787 v28787: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -24835> 2013-04-03 10:33:48.248123 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28787 update_from_paxos applying incremental 28788 -24834> 2013-04-03 10:33:48.248131 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28788 v28788: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 2045B/s wr, 0op/s -24833> 2013-04-03 10:33:48.248156 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28788 update_from_paxos applying incremental 28789 -24832> 2013-04-03 10:33:48.248164 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28789 v28789: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1843B/s wr, 0op/s -24831> 2013-04-03 10:33:48.248189 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28789 update_from_paxos applying incremental 28790 -24830> 2013-04-03 10:33:48.248197 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28790 v28790: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 424B/s wr, 1op/s -24829> 2013-04-03 10:33:48.248222 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28790 update_from_paxos applying incremental 28791 -24828> 2013-04-03 10:33:48.248233 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28791 v28791: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1967B/s wr, 0op/s -24827> 2013-04-03 10:33:48.248258 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28791 update_from_paxos applying incremental 28792 -24826> 2013-04-03 10:33:48.248268 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28792 v28792: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 3873B/s wr, 0op/s -24825> 2013-04-03 10:33:48.248294 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28792 update_from_paxos applying incremental 28793 -24824> 2013-04-03 10:33:48.248306 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28793 v28793: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1951B/s wr, 2op/s -24823> 2013-04-03 10:33:48.248340 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28793 update_from_paxos applying incremental 28794 -24822> 2013-04-03 10:33:48.248350 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28794 v28794: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1351B/s wr, 1op/s -24821> 2013-04-03 10:33:48.248376 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28794 update_from_paxos applying incremental 28795 -24820> 2013-04-03 10:33:48.248384 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28795 v28795: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1073B/s wr, 0op/s -24819> 2013-04-03 10:33:48.248409 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28795 update_from_paxos applying incremental 28796 -24818> 2013-04-03 10:33:48.248418 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28796 v28796: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s -24817> 2013-04-03 10:33:48.248443 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28796 update_from_paxos applying incremental 28797 -24816> 2013-04-03 10:33:48.248448 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28797 v28797: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1728B/s wr, 0op/s -24815> 2013-04-03 10:33:48.248473 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28797 update_from_paxos applying incremental 28798 -24814> 2013-04-03 10:33:48.248481 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28798 v28798: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail -24813> 2013-04-03 10:33:48.248823 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28798 update_from_paxos applying incremental 28799 -24812> 2013-04-03 10:33:48.248834 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28799 v28799: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 6787B/s wr, 0op/s -24811> 2013-04-03 10:33:48.248860 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28799 update_from_paxos applying incremental 28800 -24810> 2013-04-03 10:33:48.248869 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28800 v28800: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 4850B/s wr, 0op/s -24809> 2013-04-03 10:33:48.248896 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28800 update_from_paxos applying incremental 28801 -24808> 2013-04-03 10:33:48.248905 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28801 v28801: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -24807> 2013-04-03 10:33:48.248937 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28801 update_from_paxos applying incremental 28802 -24806> 2013-04-03 10:33:48.248945 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28802 v28802: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1563B/s wr, 0op/s -24805> 2013-04-03 10:33:48.248971 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28802 update_from_paxos applying incremental 28803 -24804> 2013-04-03 10:33:48.248981 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28803 v28803: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1290B/s wr, 0op/s -24803> 2013-04-03 10:33:48.249006 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28803 update_from_paxos applying incremental 28804 -24802> 2013-04-03 10:33:48.249014 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28804 v28804: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2184B/s wr, 0op/s -24801> 2013-04-03 10:33:48.249040 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28804 update_from_paxos applying incremental 28805 -24800> 2013-04-03 10:33:48.249048 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28805 v28805: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2184B/s wr, 0op/s -24799> 2013-04-03 10:33:48.249073 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28805 update_from_paxos applying incremental 28806 -24798> 2013-04-03 10:33:48.249082 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28806 v28806: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 582B/s wr, 0op/s -24797> 2013-04-03 10:33:48.249107 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28806 update_from_paxos applying incremental 28807 -24796> 2013-04-03 10:33:48.249116 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28807 v28807: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 409B/s wr, 0op/s -24795> 2013-04-03 10:33:48.249141 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28807 update_from_paxos applying incremental 28808 -24794> 2013-04-03 10:33:48.249149 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28808 v28808: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 12706B/s wr, 0op/s -24793> 2013-04-03 10:33:48.249175 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28808 update_from_paxos applying incremental 28809 -24792> 2013-04-03 10:33:48.249184 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28809 v28809: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 8191B/s wr, 0op/s -24791> 2013-04-03 10:33:48.249221 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28809 update_from_paxos applying incremental 28810 -24790> 2013-04-03 10:33:48.249231 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28810 v28810: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -24789> 2013-04-03 10:33:48.249257 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28810 update_from_paxos applying incremental 28811 -24788> 2013-04-03 10:33:48.249266 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28811 v28811: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -24787> 2013-04-03 10:33:48.249291 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28811 update_from_paxos applying incremental 28812 -24786> 2013-04-03 10:33:48.249301 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28812 v28812: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 582B/s wr, 0op/s -24785> 2013-04-03 10:33:48.249336 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28812 update_from_paxos applying incremental 28813 -24784> 2013-04-03 10:33:48.249865 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28813 v28813: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1288B/s wr, 0op/s -24783> 2013-04-03 10:33:48.249903 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28813 update_from_paxos applying incremental 28814 -24782> 2013-04-03 10:33:48.249912 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28814 v28814: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1157B/s wr, 0op/s -24781> 2013-04-03 10:33:48.249938 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28814 update_from_paxos applying incremental 28815 -24780> 2013-04-03 10:33:48.249946 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28815 v28815: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -24779> 2013-04-03 10:33:48.249972 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28815 update_from_paxos applying incremental 28816 -24778> 2013-04-03 10:33:48.249980 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28816 v28816: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2044B/s wr, 0op/s -24777> 2013-04-03 10:33:48.250007 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28816 update_from_paxos applying incremental 28817 -24776> 2013-04-03 10:33:48.250015 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28817 v28817: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1613B/s wr, 0op/s -24775> 2013-04-03 10:33:48.250048 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28817 update_from_paxos applying incremental 28818 -24774> 2013-04-03 10:33:48.250054 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28818 v28818: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 456B/s wr, 0op/s -24773> 2013-04-03 10:33:48.250081 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28818 update_from_paxos applying incremental 28819 -24772> 2013-04-03 10:33:48.250090 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28819 v28819: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1924B/s wr, 0op/s -24771> 2013-04-03 10:33:48.250116 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28819 update_from_paxos applying incremental 28820 -24770> 2013-04-03 10:33:48.250122 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28820 v28820: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1712B/s wr, 0op/s -24769> 2013-04-03 10:33:48.250146 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28820 update_from_paxos applying incremental 28821 -24768> 2013-04-03 10:33:48.250155 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28821 v28821: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 631B/s wr, 0op/s -24767> 2013-04-03 10:33:48.250180 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28821 update_from_paxos applying incremental 28822 -24766> 2013-04-03 10:33:48.250191 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28822 v28822: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 974B/s wr, 0op/s -24765> 2013-04-03 10:33:48.250217 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28822 update_from_paxos applying incremental 28823 -24764> 2013-04-03 10:33:48.250225 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28823 v28823: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3052B/s wr, 0op/s -24763> 2013-04-03 10:33:48.250250 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28823 update_from_paxos applying incremental 28824 -24762> 2013-04-03 10:33:48.250259 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28824 v28824: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2129B/s wr, 0op/s -24761> 2013-04-03 10:33:48.250285 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28824 update_from_paxos applying incremental 28825 -24760> 2013-04-03 10:33:48.250293 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28825 v28825: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 384B/s wr, 0op/s -24759> 2013-04-03 10:33:48.250318 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28825 update_from_paxos applying incremental 28826 -24758> 2013-04-03 10:33:48.250326 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28826 v28826: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 909B/s wr, 0op/s -24757> 2013-04-03 10:33:48.250351 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28826 update_from_paxos applying incremental 28827 -24756> 2013-04-03 10:33:48.250360 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28827 v28827: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -24755> 2013-04-03 10:33:48.250393 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28827 update_from_paxos applying incremental 28828 -24754> 2013-04-03 10:33:48.250401 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28828 v28828: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 582B/s wr, 0op/s -24753> 2013-04-03 10:33:48.250427 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28828 update_from_paxos applying incremental 28829 -24752> 2013-04-03 10:33:48.250435 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28829 v28829: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3046B/s wr, 0op/s -24751> 2013-04-03 10:33:48.250460 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28829 update_from_paxos applying incremental 28830 -24750> 2013-04-03 10:33:48.250468 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28830 v28830: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2129B/s wr, 0op/s -24749> 2013-04-03 10:33:48.250493 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28830 update_from_paxos applying incremental 28831 -24748> 2013-04-03 10:33:48.250503 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28831 v28831: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s -24747> 2013-04-03 10:33:48.250530 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28831 update_from_paxos applying incremental 28832 -24746> 2013-04-03 10:33:48.250541 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28832 v28832: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 15391B/s wr, 0op/s -24745> 2013-04-03 10:33:48.250567 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28832 update_from_paxos applying incremental 28833 -24744> 2013-04-03 10:33:48.250575 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28833 v28833: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 34569B/s wr, 1op/s -24743> 2013-04-03 10:33:48.250601 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28833 update_from_paxos applying incremental 28834 -24742> 2013-04-03 10:33:48.250613 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28834 v28834: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 9419B/s wr, 1op/s -24741> 2013-04-03 10:33:48.250649 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28834 update_from_paxos applying incremental 28835 -24740> 2013-04-03 10:33:48.250658 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28835 v28835: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 932B/s wr, 0op/s -24739> 2013-04-03 10:33:48.250684 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28835 update_from_paxos applying incremental 28836 -24738> 2013-04-03 10:33:48.250692 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28836 v28836: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3613B/s wr, 0op/s -24737> 2013-04-03 10:33:48.250720 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28836 update_from_paxos applying incremental 28837 -24736> 2013-04-03 10:33:48.250731 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28837 v28837: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2301B/s wr, 0op/s -24735> 2013-04-03 10:33:48.250757 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28837 update_from_paxos applying incremental 28838 -24734> 2013-04-03 10:33:48.250766 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28838 v28838: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3948B/s wr, 0op/s -24733> 2013-04-03 10:33:48.250791 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28838 update_from_paxos applying incremental 28839 -24732> 2013-04-03 10:33:48.250799 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28839 v28839: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2459B/s wr, 0op/s -24731> 2013-04-03 10:33:48.250825 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28839 update_from_paxos applying incremental 28840 -24730> 2013-04-03 10:33:48.250833 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28840 v28840: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 817B/s wr, 0op/s -24729> 2013-04-03 10:33:48.250858 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28840 update_from_paxos applying incremental 28841 -24728> 2013-04-03 10:33:48.250869 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28841 v28841: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1173B/s wr, 0op/s -24727> 2013-04-03 10:33:48.250896 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28841 update_from_paxos applying incremental 28842 -24726> 2013-04-03 10:33:48.250904 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28842 v28842: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1064B/s wr, 0op/s -24725> 2013-04-03 10:33:48.250935 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28842 update_from_paxos applying incremental 28843 -24724> 2013-04-03 10:33:48.250944 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28843 v28843: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1157B/s wr, 0op/s -24723> 2013-04-03 10:33:48.250969 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28843 update_from_paxos applying incremental 28844 -24722> 2013-04-03 10:33:48.250977 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28844 v28844: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 54244B/s wr, 0op/s -24721> 2013-04-03 10:33:48.251002 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28844 update_from_paxos applying incremental 28845 -24720> 2013-04-03 10:33:48.251010 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28845 v28845: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 100KB/s wr, 0op/s -24719> 2013-04-03 10:33:48.251035 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28845 update_from_paxos applying incremental 28846 -24718> 2013-04-03 10:33:48.251043 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28846 v28846: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1194B/s wr, 0op/s -24717> 2013-04-03 10:33:48.251068 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28846 update_from_paxos applying incremental 28847 -24716> 2013-04-03 10:33:48.251076 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28847 v28847: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1159B/s wr, 0op/s -24715> 2013-04-03 10:33:48.251101 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28847 update_from_paxos applying incremental 28848 -24714> 2013-04-03 10:33:48.251107 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28848 v28848: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 422B/s wr, 0op/s -24713> 2013-04-03 10:33:48.251131 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28848 update_from_paxos applying incremental 28849 -24712> 2013-04-03 10:33:48.251139 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28849 v28849: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 962B/s wr, 0op/s -24711> 2013-04-03 10:33:48.251164 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28849 update_from_paxos applying incremental 28850 -24710> 2013-04-03 10:33:48.251175 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28850 v28850: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1634B/s wr, 0op/s -24709> 2013-04-03 10:33:48.251201 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28850 update_from_paxos applying incremental 28851 -24708> 2013-04-03 10:33:48.251206 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28851 v28851: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 479B/s wr, 0op/s -24707> 2013-04-03 10:33:48.251230 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28851 update_from_paxos applying incremental 28852 -24706> 2013-04-03 10:33:48.251238 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28852 v28852: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 331B/s wr, 0op/s -24705> 2013-04-03 10:33:48.251270 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28852 update_from_paxos applying incremental 28853 -24704> 2013-04-03 10:33:48.251278 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28853 v28853: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 4409B/s wr, 0op/s -24703> 2013-04-03 10:33:48.251304 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28853 update_from_paxos applying incremental 28854 -24702> 2013-04-03 10:33:48.251312 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28854 v28854: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1776B/s wr, 0op/s -24701> 2013-04-03 10:33:48.251340 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28854 update_from_paxos applying incremental 28855 -24700> 2013-04-03 10:33:48.251349 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28855 v28855: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1425B/s wr, 0op/s -24699> 2013-04-03 10:33:48.251375 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28855 update_from_paxos applying incremental 28856 -24698> 2013-04-03 10:33:48.251383 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28856 v28856: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -24697> 2013-04-03 10:33:48.251408 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28856 update_from_paxos applying incremental 28857 -24696> 2013-04-03 10:33:48.251416 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28857 v28857: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 582B/s wr, 0op/s -24695> 2013-04-03 10:33:48.251442 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28857 update_from_paxos applying incremental 28858 -24694> 2013-04-03 10:33:48.251450 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28858 v28858: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3786B/s wr, 0op/s -24693> 2013-04-03 10:33:48.251476 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28858 update_from_paxos applying incremental 28859 -24692> 2013-04-03 10:33:48.251484 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28859 v28859: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2129B/s wr, 0op/s -24691> 2013-04-03 10:33:48.251509 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28859 update_from_paxos applying incremental 28860 -24690> 2013-04-03 10:33:48.251520 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28860 v28860: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -24689> 2013-04-03 10:33:48.251556 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28860 update_from_paxos applying incremental 28861 -24688> 2013-04-03 10:33:48.251566 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28861 v28861: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -24687> 2013-04-03 10:33:48.251592 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28861 update_from_paxos applying incremental 28862 -24686> 2013-04-03 10:33:48.251600 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28862 v28862: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 582B/s wr, 0op/s -24685> 2013-04-03 10:33:48.251625 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28862 update_from_paxos applying incremental 28863 -24684> 2013-04-03 10:33:48.251633 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28863 v28863: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1563B/s wr, 0op/s -24683> 2013-04-03 10:33:48.251658 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28863 update_from_paxos applying incremental 28864 -24682> 2013-04-03 10:33:48.251666 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28864 v28864: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1290B/s wr, 0op/s -24681> 2013-04-03 10:33:48.251691 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28864 update_from_paxos applying incremental 28865 -24680> 2013-04-03 10:33:48.251699 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28865 v28865: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3276B/s wr, 0op/s -24679> 2013-04-03 10:33:48.251724 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28865 update_from_paxos applying incremental 28866 -24678> 2013-04-03 10:33:48.251732 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28866 v28866: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3276B/s wr, 0op/s -24677> 2013-04-03 10:33:48.251758 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28866 update_from_paxos applying incremental 28867 -24676> 2013-04-03 10:33:48.251766 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28867 v28867: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -24675> 2013-04-03 10:33:48.251791 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28867 update_from_paxos applying incremental 28868 -24674> 2013-04-03 10:33:48.251799 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28868 v28868: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3005B/s wr, 0op/s -24673> 2013-04-03 10:33:48.251831 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28868 update_from_paxos applying incremental 28869 -24672> 2013-04-03 10:33:48.251840 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28869 v28869: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2261B/s wr, 0op/s -24671> 2013-04-03 10:33:48.251867 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28869 update_from_paxos applying incremental 28870 -24670> 2013-04-03 10:33:48.251872 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28870 v28870: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 456B/s wr, 0op/s -24669> 2013-04-03 10:33:48.251897 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28870 update_from_paxos applying incremental 28871 -24668> 2013-04-03 10:33:48.251905 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28871 v28871: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 8656B/s wr, 0op/s -24667> 2013-04-03 10:33:48.251931 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28871 update_from_paxos applying incremental 28872 -24666> 2013-04-03 10:33:48.251936 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28872 v28872: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 7689B/s wr, 0op/s -24665> 2013-04-03 10:33:48.251962 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28872 update_from_paxos applying incremental 28873 -24664> 2013-04-03 10:33:48.251970 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28873 v28873: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 631B/s wr, 0op/s -24663> 2013-04-03 10:33:48.251997 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28873 update_from_paxos applying incremental 28874 -24662> 2013-04-03 10:33:48.252005 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28874 v28874: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 977B/s wr, 0op/s -24661> 2013-04-03 10:33:48.252032 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28874 update_from_paxos applying incremental 28875 -24660> 2013-04-03 10:33:48.252040 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28875 v28875: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2043B/s wr, 0op/s -24659> 2013-04-03 10:33:48.252067 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28875 update_from_paxos applying incremental 28876 -24658> 2013-04-03 10:33:48.252075 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28876 v28876: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1613B/s wr, 0op/s -24657> 2013-04-03 10:33:48.252100 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28876 update_from_paxos applying incremental 28877 -24656> 2013-04-03 10:33:48.252108 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28877 v28877: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -24655> 2013-04-03 10:33:48.252133 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28877 update_from_paxos applying incremental 28878 -24654> 2013-04-03 10:33:48.252141 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28878 v28878: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -24653> 2013-04-03 10:33:48.252172 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28878 update_from_paxos applying incremental 28879 -24652> 2013-04-03 10:33:48.252182 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28879 v28879: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 582B/s wr, 0op/s -24651> 2013-04-03 10:33:48.252207 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28879 update_from_paxos applying incremental 28880 -24650> 2013-04-03 10:33:48.252216 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28880 v28880: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1562B/s wr, 0op/s -24649> 2013-04-03 10:33:48.252241 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28880 update_from_paxos applying incremental 28881 -24648> 2013-04-03 10:33:48.252249 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28881 v28881: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1290B/s wr, 0op/s -24647> 2013-04-03 10:33:48.252274 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28881 update_from_paxos applying incremental 28882 -24646> 2013-04-03 10:33:48.252282 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28882 v28882: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -24645> 2013-04-03 10:33:48.252308 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28882 update_from_paxos applying incremental 28883 -24644> 2013-04-03 10:33:48.252316 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28883 v28883: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -24643> 2013-04-03 10:33:48.252342 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28883 update_from_paxos applying incremental 28884 -24642> 2013-04-03 10:33:48.252350 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28884 v28884: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 582B/s wr, 0op/s -24641> 2013-04-03 10:33:48.252375 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28884 update_from_paxos applying incremental 28885 -24640> 2013-04-03 10:33:48.252383 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28885 v28885: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1562B/s wr, 0op/s -24639> 2013-04-03 10:33:48.252409 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28885 update_from_paxos applying incremental 28886 -24638> 2013-04-03 10:33:48.252417 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28886 v28886: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 920B/s wr, 0op/s -24637> 2013-04-03 10:33:48.252450 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28886 update_from_paxos applying incremental 28887 -24636> 2013-04-03 10:33:48.252459 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28887 v28887: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -24635> 2013-04-03 10:33:48.252485 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28887 update_from_paxos applying incremental 28888 -24634> 2013-04-03 10:33:48.252494 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28888 v28888: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1654B/s wr, 0op/s -24633> 2013-04-03 10:33:48.252521 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28888 update_from_paxos applying incremental 28889 -24632> 2013-04-03 10:33:48.252529 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28889 v28889: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 571B/s wr, 0op/s -24631> 2013-04-03 10:33:48.252554 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28889 update_from_paxos applying incremental 28890 -24630> 2013-04-03 10:33:48.252562 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28890 v28890: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2584B/s wr, 0op/s -24629> 2013-04-03 10:33:48.252589 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28890 update_from_paxos applying incremental 28891 -24628> 2013-04-03 10:33:48.252598 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28891 v28891: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1484B/s wr, 0op/s -24627> 2013-04-03 10:33:48.252623 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28891 update_from_paxos applying incremental 28892 -24626> 2013-04-03 10:33:48.252631 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28892 v28892: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 915B/s wr, 0op/s -24625> 2013-04-03 10:33:48.252657 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28892 update_from_paxos applying incremental 28893 -24624> 2013-04-03 10:33:48.252665 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28893 v28893: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1462B/s wr, 0op/s -24623> 2013-04-03 10:33:48.252691 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28893 update_from_paxos applying incremental 28894 -24622> 2013-04-03 10:33:48.252699 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28894 v28894: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2403B/s wr, 0op/s -24621> 2013-04-03 10:33:48.252732 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28894 update_from_paxos applying incremental 28895 -24620> 2013-04-03 10:33:48.252740 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28895 v28895: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1551B/s wr, 0op/s -24619> 2013-04-03 10:33:48.252765 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28895 update_from_paxos applying incremental 28896 -24618> 2013-04-03 10:33:48.252771 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28896 v28896: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 331B/s wr, 0op/s -24617> 2013-04-03 10:33:48.252795 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28896 update_from_paxos applying incremental 28897 -24616> 2013-04-03 10:33:48.252803 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28897 v28897: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1586B/s wr, 0op/s -24615> 2013-04-03 10:33:48.252828 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28897 update_from_paxos applying incremental 28898 -24614> 2013-04-03 10:33:48.252837 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28898 v28898: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1041B/s wr, 0op/s -24613> 2013-04-03 10:33:48.252862 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28898 update_from_paxos applying incremental 28899 -24612> 2013-04-03 10:33:48.252871 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28899 v28899: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail -24611> 2013-04-03 10:33:48.252895 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28899 update_from_paxos applying incremental 28900 -24610> 2013-04-03 10:33:48.252910 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28900 v28900: 240 pgs: 240 active+clean; 257 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1233KB/s wr, 5op/s -24609> 2013-04-03 10:33:48.252936 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28900 update_from_paxos applying incremental 28901 -24608> 2013-04-03 10:33:48.252941 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28901 v28901: 240 pgs: 240 active+clean; 257 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 850KB/s wr, 3op/s -24607> 2013-04-03 10:33:48.252966 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28901 update_from_paxos applying incremental 28902 -24606> 2013-04-03 10:33:48.252979 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28902 v28902: 240 pgs: 240 active+clean; 257 MB data, 5661 MB used, 1067 GB / 1079 GB avail -24605> 2013-04-03 10:33:48.253012 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28902 update_from_paxos applying incremental 28903 -24604> 2013-04-03 10:33:48.253020 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28903 v28903: 240 pgs: 240 active+clean; 257 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 5833B/s wr, 0op/s -24603> 2013-04-03 10:33:48.253046 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28903 update_from_paxos applying incremental 28904 -24602> 2013-04-03 10:33:48.253056 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28904 v28904: 240 pgs: 240 active+clean; 257 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 13732B/s wr, 2op/s -24601> 2013-04-03 10:33:48.253082 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28904 update_from_paxos applying incremental 28905 -24600> 2013-04-03 10:33:48.253090 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28905 v28905: 240 pgs: 240 active+clean; 257 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 3826B/s wr, 1op/s -24599> 2013-04-03 10:33:48.253115 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28905 update_from_paxos applying incremental 28906 -24598> 2013-04-03 10:33:48.253123 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28906 v28906: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 876B/s wr, 0op/s -24597> 2013-04-03 10:33:48.253149 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28906 update_from_paxos applying incremental 28907 -24596> 2013-04-03 10:33:48.253158 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28907 v28907: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 1433B/s wr, 0op/s -24595> 2013-04-03 10:33:48.253184 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28907 update_from_paxos applying incremental 28908 -24594> 2013-04-03 10:33:48.253192 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28908 v28908: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3692B/s wr, 0op/s -24593> 2013-04-03 10:33:48.253219 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28908 update_from_paxos applying incremental 28909 -24592> 2013-04-03 10:33:48.253227 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28909 v28909: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3222B/s wr, 0op/s -24591> 2013-04-03 10:33:48.253253 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28909 update_from_paxos applying incremental 28910 -24590> 2013-04-03 10:33:48.253262 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28910 v28910: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 947B/s wr, 0op/s -24589> 2013-04-03 10:33:48.253296 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28910 update_from_paxos applying incremental 28911 -24588> 2013-04-03 10:33:48.253305 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28911 v28911: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 2100B/s wr, 0op/s -24587> 2013-04-03 10:33:48.253353 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28911 update_from_paxos applying incremental 28912 -24586> 2013-04-03 10:33:48.253362 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28912 v28912: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -24585> 2013-04-03 10:33:48.253388 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28912 update_from_paxos applying incremental 28913 -24584> 2013-04-03 10:33:48.253396 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28913 v28913: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 575B/s wr, 0op/s -24583> 2013-04-03 10:33:48.253422 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28913 update_from_paxos applying incremental 28914 -24582> 2013-04-03 10:33:48.253430 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28914 v28914: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 84726B/s wr, 0op/s -24581> 2013-04-03 10:33:48.253455 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28914 update_from_paxos applying incremental 28915 -24580> 2013-04-03 10:33:48.253463 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28915 v28915: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 57620B/s wr, 0op/s -24579> 2013-04-03 10:33:48.253488 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28915 update_from_paxos applying incremental 28916 -24578> 2013-04-03 10:33:48.253496 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28916 v28916: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 356B/s wr, 0op/s -24577> 2013-04-03 10:33:48.253523 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28916 update_from_paxos applying incremental 28917 -24576> 2013-04-03 10:33:48.253533 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28917 v28917: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1442B/s wr, 0op/s -24575> 2013-04-03 10:33:48.253559 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28917 update_from_paxos applying incremental 28918 -24574> 2013-04-03 10:33:48.253567 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28918 v28918: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -24573> 2013-04-03 10:33:48.253598 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28918 update_from_paxos applying incremental 28919 -24572> 2013-04-03 10:33:48.253606 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28919 v28919: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 575B/s wr, 0op/s -24571> 2013-04-03 10:33:48.253631 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28919 update_from_paxos applying incremental 28920 -24570> 2013-04-03 10:33:48.253640 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28920 v28920: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 2043B/s wr, 0op/s -24569> 2013-04-03 10:33:48.253664 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28920 update_from_paxos applying incremental 28921 -24568> 2013-04-03 10:33:48.253672 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28921 v28921: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1627B/s wr, 0op/s -24567> 2013-04-03 10:33:48.253697 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28921 update_from_paxos applying incremental 28922 -24566> 2013-04-03 10:33:48.253705 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28922 v28922: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -24565> 2013-04-03 10:33:48.253729 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28922 update_from_paxos applying incremental 28923 -24564> 2013-04-03 10:33:48.253737 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28923 v28923: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -24563> 2013-04-03 10:33:48.253762 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28923 update_from_paxos applying incremental 28924 -24562> 2013-04-03 10:33:48.253770 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28924 v28924: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 575B/s wr, 0op/s -24561> 2013-04-03 10:33:48.253796 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28924 update_from_paxos applying incremental 28925 -24560> 2013-04-03 10:33:48.253804 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28925 v28925: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1562B/s wr, 0op/s -24559> 2013-04-03 10:33:48.253830 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28925 update_from_paxos applying incremental 28926 -24558> 2013-04-03 10:33:48.253839 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28926 v28926: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1302B/s wr, 0op/s -24557> 2013-04-03 10:33:48.253872 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28926 update_from_paxos applying incremental 28927 -24556> 2013-04-03 10:33:48.253881 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28927 v28927: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -24555> 2013-04-03 10:33:48.253907 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28927 update_from_paxos applying incremental 28928 -24554> 2013-04-03 10:33:48.253912 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28928 v28928: 240 pgs: 240 active+clean; 257 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1642B/s wr, 0op/s -24553> 2013-04-03 10:33:48.253936 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28928 update_from_paxos applying incremental 28929 -24552> 2013-04-03 10:33:48.253944 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28929 v28929: 240 pgs: 240 active+clean; 257 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 631B/s wr, 0op/s -24551> 2013-04-03 10:33:48.253977 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28929 update_from_paxos applying incremental 28930 -24550> 2013-04-03 10:33:48.253983 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28930 v28930: 240 pgs: 240 active+clean; 257 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 872B/s wr, 0op/s -24549> 2013-04-03 10:33:48.254008 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28930 update_from_paxos applying incremental 28931 -24548> 2013-04-03 10:33:48.254016 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28931 v28931: 240 pgs: 240 active+clean; 257 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 279B/s wr, 0op/s -24547> 2013-04-03 10:33:48.254042 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28931 update_from_paxos applying incremental 28932 -24546> 2013-04-03 10:33:48.254050 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28932 v28932: 240 pgs: 240 active+clean; 257 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1534B/s wr, 0op/s -24545> 2013-04-03 10:33:48.254076 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28932 update_from_paxos applying incremental 28933 -24544> 2013-04-03 10:33:48.254084 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28933 v28933: 240 pgs: 240 active+clean; 257 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1253B/s wr, 0op/s -24543> 2013-04-03 10:33:48.254110 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28933 update_from_paxos applying incremental 28934 -24542> 2013-04-03 10:33:48.254118 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28934 v28934: 240 pgs: 240 active+clean; 257 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 575B/s wr, 0op/s -24541> 2013-04-03 10:33:48.254143 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28934 update_from_paxos applying incremental 28935 -24540> 2013-04-03 10:33:48.254151 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28935 v28935: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9253B/s wr, 0op/s -24539> 2013-04-03 10:33:48.254176 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28935 update_from_paxos applying incremental 28936 -24538> 2013-04-03 10:33:48.254185 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28936 v28936: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 6510B/s wr, 0op/s -24537> 2013-04-03 10:33:48.254222 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28936 update_from_paxos applying incremental 28937 -24536> 2013-04-03 10:33:48.254232 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28937 v28937: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2866B/s wr, 0op/s -24535> 2013-04-03 10:33:48.254259 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28937 update_from_paxos applying incremental 28938 -24534> 2013-04-03 10:33:48.254267 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28938 v28938: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2866B/s wr, 0op/s -24533> 2013-04-03 10:33:48.254293 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28938 update_from_paxos applying incremental 28939 -24532> 2013-04-03 10:33:48.254301 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28939 v28939: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 575B/s wr, 0op/s -24531> 2013-04-03 10:33:48.254327 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28939 update_from_paxos applying incremental 28940 -24530> 2013-04-03 10:33:48.254335 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28940 v28940: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4256B/s wr, 0op/s -24529> 2013-04-03 10:33:48.254360 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28940 update_from_paxos applying incremental 28941 -24528> 2013-04-03 10:33:48.254368 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28941 v28941: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 3627B/s wr, 0op/s -24527> 2013-04-03 10:33:48.254393 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28941 update_from_paxos applying incremental 28942 -24526> 2013-04-03 10:33:48.254401 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28942 v28942: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 575B/s wr, 0op/s -24525> 2013-04-03 10:33:48.254427 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28942 update_from_paxos applying incremental 28943 -24524> 2013-04-03 10:33:48.254435 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28943 v28943: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 3792B/s wr, 0op/s -24523> 2013-04-03 10:33:48.254462 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28943 update_from_paxos applying incremental 28944 -24522> 2013-04-03 10:33:48.254469 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28944 v28944: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2160B/s wr, 0op/s -24521> 2013-04-03 10:33:48.254504 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28944 update_from_paxos applying incremental 28945 -24520> 2013-04-03 10:33:48.254513 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28945 v28945: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2454B/s wr, 0op/s -24519> 2013-04-03 10:33:48.254539 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28945 update_from_paxos applying incremental 28946 -24518> 2013-04-03 10:33:48.254548 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28946 v28946: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2457B/s wr, 0op/s -24517> 2013-04-03 10:33:48.254573 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28946 update_from_paxos applying incremental 28947 -24516> 2013-04-03 10:33:48.254581 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28947 v28947: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 574B/s wr, 0op/s -24515> 2013-04-03 10:33:48.254607 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28947 update_from_paxos applying incremental 28948 -24514> 2013-04-03 10:33:48.254615 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28948 v28948: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1287B/s wr, 0op/s -24513> 2013-04-03 10:33:48.254640 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28948 update_from_paxos applying incremental 28949 -24512> 2013-04-03 10:33:48.254648 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28949 v28949: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1223B/s wr, 0op/s -24511> 2013-04-03 10:33:48.254673 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28949 update_from_paxos applying incremental 28950 -24510> 2013-04-03 10:33:48.254681 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28950 v28950: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 690B/s wr, 0op/s -24509> 2013-04-03 10:33:48.254707 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28950 update_from_paxos applying incremental 28951 -24508> 2013-04-03 10:33:48.254712 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28951 v28951: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 136B/s wr, 0op/s -24507> 2013-04-03 10:33:48.254738 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28951 update_from_paxos applying incremental 28952 -24506> 2013-04-03 10:33:48.254746 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28952 v28952: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1982B/s wr, 0op/s -24505> 2013-04-03 10:33:48.254771 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28952 update_from_paxos applying incremental 28953 -24504> 2013-04-03 10:33:48.254776 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28953 v28953: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 2661B/s wr, 0op/s -24503> 2013-04-03 10:33:48.254801 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28953 update_from_paxos applying incremental 28954 -24502> 2013-04-03 10:33:48.254809 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28954 v28954: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 641B/s wr, 0op/s -24501> 2013-04-03 10:33:48.254873 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28954 update_from_paxos applying incremental 28955 -24500> 2013-04-03 10:33:48.254881 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28955 v28955: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 438B/s wr, 0op/s -24499> 2013-04-03 10:33:48.254911 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28955 update_from_paxos applying incremental 28956 -24498> 2013-04-03 10:33:48.255079 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28956 v28956: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail -24497> 2013-04-03 10:33:48.255155 7fce0612e780 7 mon.hotcpc9039@-1(probing).pg v28956 update_from_paxos applying incremental 28957 -24496> 2013-04-03 10:33:48.255313 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28957 v28957: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 4872B/s wr, 0op/s -24495> 2013-04-03 10:33:48.255351 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28957 send_pg_creates to 0 pgs -24494> 2013-04-03 10:33:48.255357 7fce0612e780 10 mon.hotcpc9039@-1(probing).pg v28957 update_logger -24493> 2013-04-03 10:33:48.255389 7fce0612e780 10 mon.hotcpc9039@-1(probing).mds e0 update_from_paxos version 184, my e 0 -24492> 2013-04-03 10:33:48.255424 7fce0612e780 10 mon.hotcpc9039@-1(probing).mds e0 update_from_paxos got 184 -24491> 2013-04-03 10:33:48.255446 7fce0612e780 4 mon.hotcpc9039@-1(probing).mds e184 new map -24490> 2013-04-03 10:33:48.255451 7fce0612e780 7 mon.hotcpc9039@-1(probing).mds e184 print_map epoch 184 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:33:42.867413 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 596 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:replay seq 1 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11006: 10.252.24.88:6800/17213 'hotcpc12533' mds.-1.0 up:standby seq 1 -24489> 2013-04-03 10:33:48.255481 7fce0612e780 10 mon.hotcpc9039@-1(probing).mds e184 update_logger -24488> 2013-04-03 10:33:48.255694 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e0 update_from_paxos loading latest full map e379 -24487> 2013-04-03 10:33:48.255798 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e379 update_from_paxos applying incremental 380 -24486> 2013-04-03 10:33:48.255845 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e380 e380: 3 osds: 2 up, 3 in -24485> 2013-04-03 10:33:48.255876 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e380 update_from_paxos applying incremental 381 -24484> 2013-04-03 10:33:48.255914 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e381 e381: 3 osds: 3 up, 3 in -24483> 2013-04-03 10:33:48.255937 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e381 update_from_paxos applying incremental 382 -24482> 2013-04-03 10:33:48.255965 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e382 e382: 3 osds: 3 up, 3 in -24481> 2013-04-03 10:33:48.255989 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e382 update_from_paxos applying incremental 383 -24480> 2013-04-03 10:33:48.256017 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e383 e383: 3 osds: 3 up, 3 in -24479> 2013-04-03 10:33:48.256039 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e383 update_from_paxos applying incremental 384 -24478> 2013-04-03 10:33:48.256074 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e384 e384: 3 osds: 3 up, 3 in -24477> 2013-04-03 10:33:48.256100 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e384 update_from_paxos applying incremental 385 -24476> 2013-04-03 10:33:48.256133 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e385 e385: 3 osds: 2 up, 3 in -24475> 2013-04-03 10:33:48.256159 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e385 update_from_paxos applying incremental 386 -24474> 2013-04-03 10:33:48.256195 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e386 e386: 3 osds: 3 up, 3 in -24473> 2013-04-03 10:33:48.256226 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e386 update_from_paxos applying incremental 387 -24472> 2013-04-03 10:33:48.256261 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e387 e387: 3 osds: 3 up, 3 in -24471> 2013-04-03 10:33:48.256290 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e387 update_from_paxos applying incremental 388 -24470> 2013-04-03 10:33:48.256325 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e388 e388: 3 osds: 3 up, 3 in -24469> 2013-04-03 10:33:48.256349 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e388 update_from_paxos applying incremental 389 -24468> 2013-04-03 10:33:48.256402 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e389 e389: 3 osds: 3 up, 3 in -24467> 2013-04-03 10:33:48.256436 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e389 update_from_paxos applying incremental 390 -24466> 2013-04-03 10:33:48.256482 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e390 e390: 3 osds: 3 up, 3 in -24465> 2013-04-03 10:33:48.256513 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e390 update_from_paxos applying incremental 391 -24464> 2013-04-03 10:33:48.256546 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e391 e391: 3 osds: 3 up, 3 in -24463> 2013-04-03 10:33:48.256569 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e391 update_from_paxos applying incremental 392 -24462> 2013-04-03 10:33:48.256610 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e392 e392: 3 osds: 3 up, 3 in -24461> 2013-04-03 10:33:48.256633 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e392 update_from_paxos applying incremental 393 -24460> 2013-04-03 10:33:48.256666 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e393 e393: 3 osds: 3 up, 3 in -24459> 2013-04-03 10:33:48.256688 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e393 update_from_paxos applying incremental 394 -24458> 2013-04-03 10:33:48.256725 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e394 e394: 3 osds: 3 up, 3 in -24457> 2013-04-03 10:33:48.256748 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e394 update_from_paxos applying incremental 395 -24456> 2013-04-03 10:33:48.256782 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e395 e395: 3 osds: 3 up, 3 in -24455> 2013-04-03 10:33:48.256804 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e395 update_from_paxos applying incremental 396 -24454> 2013-04-03 10:33:48.256838 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e396 e396: 3 osds: 3 up, 3 in -24453> 2013-04-03 10:33:48.256860 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e396 update_from_paxos applying incremental 397 -24452> 2013-04-03 10:33:48.256894 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e397 e397: 3 osds: 3 up, 3 in -24451> 2013-04-03 10:33:48.256916 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e397 update_from_paxos applying incremental 398 -24450> 2013-04-03 10:33:48.256968 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e398 e398: 3 osds: 3 up, 3 in -24449> 2013-04-03 10:33:48.257168 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e398 update_from_paxos applying incremental 399 -24448> 2013-04-03 10:33:48.257269 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e399 e399: 3 osds: 3 up, 3 in -24447> 2013-04-03 10:33:48.257320 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e399 update_from_paxos applying incremental 400 -24446> 2013-04-03 10:33:48.257516 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e400 e400: 3 osds: 3 up, 3 in -24445> 2013-04-03 10:33:48.257574 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e400 update_from_paxos applying incremental 401 -24444> 2013-04-03 10:33:48.257657 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e401 e401: 3 osds: 3 up, 3 in -24443> 2013-04-03 10:33:48.257683 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e401 update_from_paxos applying incremental 402 -24442> 2013-04-03 10:33:48.257783 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e402 e402: 3 osds: 3 up, 3 in -24441> 2013-04-03 10:33:48.257808 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e402 update_from_paxos applying incremental 403 -24440> 2013-04-03 10:33:48.257895 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e403 e403: 3 osds: 3 up, 3 in -24439> 2013-04-03 10:33:48.257920 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e403 update_from_paxos applying incremental 404 -24438> 2013-04-03 10:33:48.258005 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e404 e404: 3 osds: 3 up, 3 in -24437> 2013-04-03 10:33:48.258039 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e404 update_from_paxos applying incremental 405 -24436> 2013-04-03 10:33:48.258141 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e405 e405: 3 osds: 3 up, 3 in -24435> 2013-04-03 10:33:48.258184 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e405 update_from_paxos applying incremental 406 -24434> 2013-04-03 10:33:48.258216 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e406 e406: 3 osds: 3 up, 3 in -24433> 2013-04-03 10:33:48.258239 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e406 update_from_paxos applying incremental 407 -24432> 2013-04-03 10:33:48.258269 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e407 e407: 3 osds: 3 up, 3 in -24431> 2013-04-03 10:33:48.258292 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e407 update_from_paxos applying incremental 408 -24430> 2013-04-03 10:33:48.258321 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e408 e408: 3 osds: 3 up, 3 in -24429> 2013-04-03 10:33:48.258342 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e408 update_from_paxos applying incremental 409 -24428> 2013-04-03 10:33:48.258371 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e409 e409: 3 osds: 3 up, 3 in -24427> 2013-04-03 10:33:48.258393 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e409 update_from_paxos applying incremental 410 -24426> 2013-04-03 10:33:48.258421 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e410 e410: 3 osds: 3 up, 3 in -24425> 2013-04-03 10:33:48.258443 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e410 update_from_paxos applying incremental 411 -24424> 2013-04-03 10:33:48.258492 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e411 e411: 3 osds: 3 up, 3 in -24423> 2013-04-03 10:33:48.258514 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e411 update_from_paxos applying incremental 412 -24422> 2013-04-03 10:33:48.258547 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e412 e412: 3 osds: 3 up, 3 in -24421> 2013-04-03 10:33:48.258577 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e412 update_from_paxos applying incremental 413 -24420> 2013-04-03 10:33:48.258605 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e413 e413: 3 osds: 3 up, 3 in -24419> 2013-04-03 10:33:48.258628 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e413 update_from_paxos applying incremental 414 -24418> 2013-04-03 10:33:48.258655 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e414 e414: 3 osds: 3 up, 3 in -24417> 2013-04-03 10:33:48.258678 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e414 update_from_paxos applying incremental 415 -24416> 2013-04-03 10:33:48.258705 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e415 e415: 3 osds: 3 up, 3 in -24415> 2013-04-03 10:33:48.258729 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e415 update_from_paxos applying incremental 416 -24414> 2013-04-03 10:33:48.258755 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e416 e416: 3 osds: 3 up, 3 in -24413> 2013-04-03 10:33:48.258778 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e416 update_from_paxos applying incremental 417 -24412> 2013-04-03 10:33:48.258805 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e417 e417: 3 osds: 3 up, 3 in -24411> 2013-04-03 10:33:48.258928 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e417 update_from_paxos applying incremental 418 -24410> 2013-04-03 10:33:48.258959 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e418 e418: 3 osds: 3 up, 3 in -24409> 2013-04-03 10:33:48.258983 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e418 update_from_paxos applying incremental 419 -24408> 2013-04-03 10:33:48.259008 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e419 e419: 3 osds: 2 up, 3 in -24407> 2013-04-03 10:33:48.259031 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e419 update_from_paxos applying incremental 420 -24406> 2013-04-03 10:33:48.259057 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e420 e420: 3 osds: 2 up, 3 in -24405> 2013-04-03 10:33:48.259080 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e420 update_from_paxos applying incremental 421 -24404> 2013-04-03 10:33:48.259106 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e421 e421: 3 osds: 2 up, 3 in -24403> 2013-04-03 10:33:48.259128 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e421 update_from_paxos applying incremental 422 -24402> 2013-04-03 10:33:48.259156 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e422 e422: 3 osds: 3 up, 3 in -24401> 2013-04-03 10:33:48.259193 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e422 update_from_paxos applying incremental 423 -24400> 2013-04-03 10:33:48.259221 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e423 e423: 3 osds: 3 up, 3 in -24399> 2013-04-03 10:33:48.259245 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e423 update_from_paxos applying incremental 424 -24398> 2013-04-03 10:33:48.259270 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e424 e424: 4 osds: 3 up, 3 in -24397> 2013-04-03 10:33:48.259291 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e424 update_from_paxos applying incremental 425 -24396> 2013-04-03 10:33:48.259323 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e425 e425: 5 osds: 3 up, 3 in -24395> 2013-04-03 10:33:48.259346 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e425 update_from_paxos applying incremental 426 -24394> 2013-04-03 10:33:48.259375 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e426 e426: 5 osds: 4 up, 4 in -24393> 2013-04-03 10:33:48.259398 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e426 update_from_paxos applying incremental 427 -24392> 2013-04-03 10:33:48.259427 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e427 e427: 5 osds: 5 up, 5 in -24391> 2013-04-03 10:33:48.259455 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e427 update_from_paxos applying incremental 428 -24390> 2013-04-03 10:33:48.259501 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e428 e428: 5 osds: 5 up, 5 in -24389> 2013-04-03 10:33:48.259524 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e428 update_from_paxos applying incremental 429 -24388> 2013-04-03 10:33:48.259554 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e429 e429: 5 osds: 5 up, 5 in -24387> 2013-04-03 10:33:48.259577 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e429 update_from_paxos applying incremental 430 -24386> 2013-04-03 10:33:48.259606 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e430 e430: 5 osds: 5 up, 5 in -24385> 2013-04-03 10:33:48.259628 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e430 update_from_paxos applying incremental 431 -24384> 2013-04-03 10:33:48.259654 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e431 e431: 5 osds: 5 up, 5 in -24383> 2013-04-03 10:33:48.259677 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e431 update_from_paxos applying incremental 432 -24382> 2013-04-03 10:33:48.259707 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e432 e432: 5 osds: 5 up, 5 in -24381> 2013-04-03 10:33:48.259730 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e432 update_from_paxos applying incremental 433 -24380> 2013-04-03 10:33:48.259757 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e433 e433: 5 osds: 5 up, 5 in -24379> 2013-04-03 10:33:48.259779 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e433 update_from_paxos applying incremental 434 -24378> 2013-04-03 10:33:48.259808 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e434 e434: 5 osds: 5 up, 5 in -24377> 2013-04-03 10:33:48.259837 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e434 update_from_paxos applying incremental 435 -24376> 2013-04-03 10:33:48.259865 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e435 e435: 5 osds: 5 up, 5 in -24375> 2013-04-03 10:33:48.259888 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e435 update_from_paxos applying incremental 436 -24374> 2013-04-03 10:33:48.259915 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e436 e436: 5 osds: 5 up, 5 in -24373> 2013-04-03 10:33:48.259936 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e436 update_from_paxos applying incremental 437 -24372> 2013-04-03 10:33:48.259964 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e437 e437: 5 osds: 5 up, 5 in -24371> 2013-04-03 10:33:48.259997 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e437 update_from_paxos applying incremental 438 -24370> 2013-04-03 10:33:48.260025 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e438 e438: 5 osds: 5 up, 5 in -24369> 2013-04-03 10:33:48.260048 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e438 update_from_paxos applying incremental 439 -24368> 2013-04-03 10:33:48.260076 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e439 e439: 5 osds: 5 up, 5 in -24367> 2013-04-03 10:33:48.260099 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e439 update_from_paxos applying incremental 440 -24366> 2013-04-03 10:33:48.260127 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e440 e440: 5 osds: 5 up, 5 in -24365> 2013-04-03 10:33:48.260149 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e440 update_from_paxos applying incremental 441 -24364> 2013-04-03 10:33:48.260177 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e441 e441: 5 osds: 5 up, 5 in -24363> 2013-04-03 10:33:48.260199 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e441 update_from_paxos applying incremental 442 -24362> 2013-04-03 10:33:48.260226 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e442 e442: 5 osds: 5 up, 5 in -24361> 2013-04-03 10:33:48.260248 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e442 update_from_paxos applying incremental 443 -24360> 2013-04-03 10:33:48.260296 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e443 e443: 5 osds: 5 up, 5 in -24359> 2013-04-03 10:33:48.260330 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e443 update_from_paxos applying incremental 444 -24358> 2013-04-03 10:33:48.260359 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e444 e444: 5 osds: 4 up, 5 in -24357> 2013-04-03 10:33:48.260382 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e444 update_from_paxos applying incremental 445 -24356> 2013-04-03 10:33:48.260478 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e445 e445: 5 osds: 5 up, 5 in -24355> 2013-04-03 10:33:48.260524 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e445 update_from_paxos applying incremental 446 -24354> 2013-04-03 10:33:48.260589 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e446 e446: 5 osds: 5 up, 5 in -24353> 2013-04-03 10:33:48.260625 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e446 update_from_paxos applying incremental 447 -24352> 2013-04-03 10:33:48.260655 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e447 e447: 5 osds: 5 up, 5 in -24351> 2013-04-03 10:33:48.260678 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e447 update_from_paxos applying incremental 448 -24350> 2013-04-03 10:33:48.260707 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e448 e448: 5 osds: 5 up, 5 in -24349> 2013-04-03 10:33:48.260730 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e448 update_from_paxos applying incremental 449 -24348> 2013-04-03 10:33:48.260758 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e449 e449: 5 osds: 5 up, 5 in -24347> 2013-04-03 10:33:48.260781 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e449 update_from_paxos applying incremental 450 -24346> 2013-04-03 10:33:48.260809 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e450 e450: 5 osds: 5 up, 5 in -24345> 2013-04-03 10:33:48.260831 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e450 update_from_paxos applying incremental 451 -24344> 2013-04-03 10:33:48.260859 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e451 e451: 5 osds: 4 up, 5 in -24343> 2013-04-03 10:33:48.260880 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e451 update_from_paxos applying incremental 452 -24342> 2013-04-03 10:33:48.260933 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e452 e452: 5 osds: 5 up, 5 in -24341> 2013-04-03 10:33:48.260971 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e452 update_from_paxos applying incremental 453 -24340> 2013-04-03 10:33:48.261001 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e453 e453: 5 osds: 5 up, 5 in -24339> 2013-04-03 10:33:48.261024 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e453 update_from_paxos applying incremental 454 -24338> 2013-04-03 10:33:48.261051 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e454 e454: 5 osds: 5 up, 5 in -24337> 2013-04-03 10:33:48.261074 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e454 update_from_paxos applying incremental 455 -24336> 2013-04-03 10:33:48.261102 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e455 e455: 5 osds: 5 up, 5 in -24335> 2013-04-03 10:33:48.261124 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e455 update_from_paxos applying incremental 456 -24334> 2013-04-03 10:33:48.261152 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e456 e456: 5 osds: 5 up, 5 in -24333> 2013-04-03 10:33:48.261174 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e456 update_from_paxos applying incremental 457 -24332> 2013-04-03 10:33:48.261223 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e457 e457: 5 osds: 5 up, 5 in -24331> 2013-04-03 10:33:48.261253 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e457 update_from_paxos applying incremental 458 -24330> 2013-04-03 10:33:48.261284 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e458 e458: 5 osds: 5 up, 5 in -24329> 2013-04-03 10:33:48.261308 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e458 update_from_paxos applying incremental 459 -24328> 2013-04-03 10:33:48.261345 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e459 e459: 5 osds: 5 up, 5 in -24327> 2013-04-03 10:33:48.261368 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e459 update_from_paxos applying incremental 460 -24326> 2013-04-03 10:33:48.261397 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e460 e460: 5 osds: 5 up, 5 in -24325> 2013-04-03 10:33:48.261419 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e460 update_from_paxos applying incremental 461 -24324> 2013-04-03 10:33:48.261447 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e461 e461: 5 osds: 5 up, 5 in -24323> 2013-04-03 10:33:48.261469 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e461 update_from_paxos applying incremental 462 -24322> 2013-04-03 10:33:48.261498 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e462 e462: 5 osds: 5 up, 5 in -24321> 2013-04-03 10:33:48.261520 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e462 update_from_paxos applying incremental 463 -24320> 2013-04-03 10:33:48.261548 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e463 e463: 5 osds: 5 up, 5 in -24319> 2013-04-03 10:33:48.261570 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e463 update_from_paxos applying incremental 464 -24318> 2013-04-03 10:33:48.261598 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e464 e464: 5 osds: 5 up, 5 in -24317> 2013-04-03 10:33:48.261620 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e464 update_from_paxos applying incremental 465 -24316> 2013-04-03 10:33:48.261648 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e465 e465: 5 osds: 5 up, 5 in -24315> 2013-04-03 10:33:48.261772 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e465 update_from_paxos applying incremental 466 -24314> 2013-04-03 10:33:48.261830 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e466 e466: 5 osds: 5 up, 5 in -24313> 2013-04-03 10:33:48.261863 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e466 update_from_paxos applying incremental 467 -24312> 2013-04-03 10:33:48.261895 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e467 e467: 5 osds: 5 up, 5 in -24311> 2013-04-03 10:33:48.261918 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e467 update_from_paxos applying incremental 468 -24310> 2013-04-03 10:33:48.261947 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e468 e468: 5 osds: 5 up, 5 in -24309> 2013-04-03 10:33:48.261969 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e468 update_from_paxos applying incremental 469 -24308> 2013-04-03 10:33:48.261997 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e469 e469: 5 osds: 5 up, 5 in -24307> 2013-04-03 10:33:48.262019 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e469 update_from_paxos applying incremental 470 -24306> 2013-04-03 10:33:48.262047 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e470 e470: 5 osds: 5 up, 5 in -24305> 2013-04-03 10:33:48.262068 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e470 update_from_paxos applying incremental 471 -24304> 2013-04-03 10:33:48.262096 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e471 e471: 5 osds: 5 up, 5 in -24303> 2013-04-03 10:33:48.262123 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e471 update_from_paxos applying incremental 472 -24302> 2013-04-03 10:33:48.262151 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e472 e472: 5 osds: 5 up, 5 in -24301> 2013-04-03 10:33:48.262174 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e472 update_from_paxos applying incremental 473 -24300> 2013-04-03 10:33:48.262201 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e473 e473: 5 osds: 4 up, 5 in -24299> 2013-04-03 10:33:48.262223 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e473 update_from_paxos applying incremental 474 -24298> 2013-04-03 10:33:48.262250 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e474 e474: 5 osds: 4 up, 5 in -24297> 2013-04-03 10:33:48.262272 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e474 update_from_paxos applying incremental 475 -24296> 2013-04-03 10:33:48.262299 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e475 e475: 5 osds: 3 up, 5 in -24295> 2013-04-03 10:33:48.262321 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e475 update_from_paxos applying incremental 476 -24294> 2013-04-03 10:33:48.262349 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e476 e476: 5 osds: 3 up, 5 in -24293> 2013-04-03 10:33:48.262370 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e476 update_from_paxos applying incremental 477 -24292> 2013-04-03 10:33:48.262419 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e477 e477: 5 osds: 3 up, 5 in -24291> 2013-04-03 10:33:48.262450 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e477 update_from_paxos applying incremental 478 -24290> 2013-04-03 10:33:48.262493 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e478 e478: 5 osds: 3 up, 5 in -24289> 2013-04-03 10:33:48.262517 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e478 update_from_paxos applying incremental 479 -24288> 2013-04-03 10:33:48.262552 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e479 e479: 5 osds: 4 up, 5 in -24287> 2013-04-03 10:33:48.262589 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e479 update_from_paxos applying incremental 480 -24286> 2013-04-03 10:33:48.262642 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e480 e480: 5 osds: 4 up, 5 in -24285> 2013-04-03 10:33:48.262675 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e480 update_from_paxos applying incremental 481 -24284> 2013-04-03 10:33:48.263162 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e481 e481: 5 osds: 4 up, 5 in -24283> 2013-04-03 10:33:48.263192 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e481 update_from_paxos applying incremental 482 -24282> 2013-04-03 10:33:48.263233 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e482 e482: 5 osds: 4 up, 5 in -24281> 2013-04-03 10:33:48.263257 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e482 update_from_paxos applying incremental 483 -24280> 2013-04-03 10:33:48.263295 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e483 e483: 5 osds: 4 up, 5 in -24279> 2013-04-03 10:33:48.263318 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e483 update_from_paxos applying incremental 484 -24278> 2013-04-03 10:33:48.263378 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e484 e484: 5 osds: 4 up, 5 in -24277> 2013-04-03 10:33:48.263410 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e484 update_from_paxos applying incremental 485 -24276> 2013-04-03 10:33:48.263507 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e485 e485: 5 osds: 4 up, 5 in -24275> 2013-04-03 10:33:48.263548 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e485 update_from_paxos applying incremental 486 -24274> 2013-04-03 10:33:48.263607 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e486 e486: 5 osds: 4 up, 5 in -24273> 2013-04-03 10:33:48.263631 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e486 update_from_paxos applying incremental 487 -24272> 2013-04-03 10:33:48.263694 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e487 e487: 5 osds: 5 up, 5 in -24271> 2013-04-03 10:33:48.263719 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e487 update_from_paxos applying incremental 488 -24270> 2013-04-03 10:33:48.263785 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e488 e488: 5 osds: 5 up, 5 in -24269> 2013-04-03 10:33:48.263816 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e488 update_from_paxos applying incremental 489 -24268> 2013-04-03 10:33:48.263846 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e489 e489: 5 osds: 5 up, 5 in -24267> 2013-04-03 10:33:48.263870 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e489 update_from_paxos applying incremental 490 -24266> 2013-04-03 10:33:48.263899 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e490 e490: 5 osds: 5 up, 5 in -24265> 2013-04-03 10:33:48.263922 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e490 update_from_paxos applying incremental 491 -24264> 2013-04-03 10:33:48.263950 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e491 e491: 5 osds: 5 up, 5 in -24263> 2013-04-03 10:33:48.263971 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e491 update_from_paxos applying incremental 492 -24262> 2013-04-03 10:33:48.263999 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e492 e492: 5 osds: 5 up, 5 in -24261> 2013-04-03 10:33:48.264020 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e492 update_from_paxos applying incremental 493 -24260> 2013-04-03 10:33:48.264047 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e493 e493: 5 osds: 5 up, 5 in -24259> 2013-04-03 10:33:48.264069 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e493 update_from_paxos applying incremental 494 -24258> 2013-04-03 10:33:48.264097 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e494 e494: 5 osds: 4 up, 5 in -24257> 2013-04-03 10:33:48.264129 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e494 update_from_paxos applying incremental 495 -24256> 2013-04-03 10:33:48.264158 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e495 e495: 5 osds: 4 up, 5 in -24255> 2013-04-03 10:33:48.264180 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e495 update_from_paxos applying incremental 496 -24254> 2013-04-03 10:33:48.264209 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e496 e496: 5 osds: 5 up, 5 in -24253> 2013-04-03 10:33:48.264232 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e496 update_from_paxos applying incremental 497 -24252> 2013-04-03 10:33:48.264260 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e497 e497: 5 osds: 5 up, 5 in -24251> 2013-04-03 10:33:48.264290 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e497 update_from_paxos applying incremental 498 -24250> 2013-04-03 10:33:48.264319 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e498 e498: 5 osds: 5 up, 5 in -24249> 2013-04-03 10:33:48.264341 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e498 update_from_paxos applying incremental 499 -24248> 2013-04-03 10:33:48.264391 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e499 e499: 5 osds: 4 up, 5 in -24247> 2013-04-03 10:33:48.264415 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e499 update_from_paxos applying incremental 500 -24246> 2013-04-03 10:33:48.264498 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e500 e500: 5 osds: 5 up, 5 in -24245> 2013-04-03 10:33:48.264540 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e500 update_from_paxos applying incremental 501 -24244> 2013-04-03 10:33:48.264600 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e501 e501: 5 osds: 5 up, 5 in -24243> 2013-04-03 10:33:48.264636 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e501 update_from_paxos applying incremental 502 -24242> 2013-04-03 10:33:48.264682 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e502 e502: 5 osds: 5 up, 5 in -24241> 2013-04-03 10:33:48.264706 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e502 update_from_paxos applying incremental 503 -24240> 2013-04-03 10:33:48.264741 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e503 e503: 5 osds: 5 up, 5 in -24239> 2013-04-03 10:33:48.264763 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e503 update_from_paxos applying incremental 504 -24238> 2013-04-03 10:33:48.264797 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e504 e504: 5 osds: 5 up, 5 in -24237> 2013-04-03 10:33:48.264836 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e504 update_from_paxos applying incremental 505 -24236> 2013-04-03 10:33:48.264872 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e505 e505: 5 osds: 5 up, 5 in -24235> 2013-04-03 10:33:48.264896 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e505 update_from_paxos applying incremental 506 -24234> 2013-04-03 10:33:48.264930 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e506 e506: 5 osds: 5 up, 5 in -24233> 2013-04-03 10:33:48.264954 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e506 update_from_paxos applying incremental 507 -24232> 2013-04-03 10:33:48.264991 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e507 e507: 5 osds: 5 up, 5 in -24231> 2013-04-03 10:33:48.265015 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e507 update_from_paxos applying incremental 508 -24230> 2013-04-03 10:33:48.265048 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e508 e508: 5 osds: 5 up, 5 in -24229> 2013-04-03 10:33:48.265072 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e508 update_from_paxos applying incremental 509 -24228> 2013-04-03 10:33:48.265105 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e509 e509: 5 osds: 5 up, 5 in -24227> 2013-04-03 10:33:48.265136 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e509 update_from_paxos applying incremental 510 -24226> 2013-04-03 10:33:48.265172 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e510 e510: 5 osds: 5 up, 5 in -24225> 2013-04-03 10:33:48.265199 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e510 update_from_paxos applying incremental 511 -24224> 2013-04-03 10:33:48.265231 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e511 e511: 5 osds: 5 up, 5 in -24223> 2013-04-03 10:33:48.265254 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e511 update_from_paxos applying incremental 512 -24222> 2013-04-03 10:33:48.265287 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e512 e512: 5 osds: 5 up, 5 in -24221> 2013-04-03 10:33:48.265310 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e512 update_from_paxos applying incremental 513 -24220> 2013-04-03 10:33:48.265351 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e513 e513: 5 osds: 5 up, 5 in -24219> 2013-04-03 10:33:48.265382 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e513 update_from_paxos applying incremental 514 -24218> 2013-04-03 10:33:48.265416 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e514 e514: 5 osds: 5 up, 5 in -24217> 2013-04-03 10:33:48.265439 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e514 update_from_paxos applying incremental 515 -24216> 2013-04-03 10:33:48.265471 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e515 e515: 5 osds: 4 up, 5 in -24215> 2013-04-03 10:33:48.265498 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e515 update_from_paxos applying incremental 516 -24214> 2013-04-03 10:33:48.265533 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e516 e516: 5 osds: 5 up, 5 in -24213> 2013-04-03 10:33:48.265557 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e516 update_from_paxos applying incremental 517 -24212> 2013-04-03 10:33:48.265589 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e517 e517: 5 osds: 5 up, 5 in -24211> 2013-04-03 10:33:48.265613 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e517 update_from_paxos applying incremental 518 -24210> 2013-04-03 10:33:48.265664 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e518 e518: 5 osds: 5 up, 5 in -24209> 2013-04-03 10:33:48.265693 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e518 update_from_paxos applying incremental 519 -24208> 2013-04-03 10:33:48.265732 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e519 e519: 5 osds: 5 up, 5 in -24207> 2013-04-03 10:33:48.265757 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e519 update_from_paxos applying incremental 520 -24206> 2013-04-03 10:33:48.265791 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e520 e520: 5 osds: 5 up, 5 in -24205> 2013-04-03 10:33:48.265815 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e520 update_from_paxos applying incremental 521 -24204> 2013-04-03 10:33:48.265849 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e521 e521: 5 osds: 5 up, 5 in -24203> 2013-04-03 10:33:48.265874 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e521 update_from_paxos applying incremental 522 -24202> 2013-04-03 10:33:48.265907 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e522 e522: 5 osds: 5 up, 5 in -24201> 2013-04-03 10:33:48.265931 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e522 update_from_paxos applying incremental 523 -24200> 2013-04-03 10:33:48.265966 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e523 e523: 5 osds: 5 up, 5 in -24199> 2013-04-03 10:33:48.265991 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e523 update_from_paxos applying incremental 524 -24198> 2013-04-03 10:33:48.266024 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e524 e524: 5 osds: 5 up, 5 in -24197> 2013-04-03 10:33:48.266047 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e524 update_from_paxos applying incremental 525 -24196> 2013-04-03 10:33:48.266086 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e525 e525: 5 osds: 5 up, 5 in -24195> 2013-04-03 10:33:48.266121 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e525 update_from_paxos applying incremental 526 -24194> 2013-04-03 10:33:48.266154 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e526 e526: 5 osds: 5 up, 5 in -24193> 2013-04-03 10:33:48.266177 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e526 update_from_paxos applying incremental 527 -24192> 2013-04-03 10:33:48.266205 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e527 e527: 5 osds: 5 up, 5 in -24191> 2013-04-03 10:33:48.266227 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e527 update_from_paxos applying incremental 528 -24190> 2013-04-03 10:33:48.266254 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e528 e528: 5 osds: 5 up, 5 in -24189> 2013-04-03 10:33:48.266281 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e528 update_from_paxos applying incremental 529 -24188> 2013-04-03 10:33:48.266309 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e529 e529: 5 osds: 5 up, 5 in -24187> 2013-04-03 10:33:48.266331 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e529 update_from_paxos applying incremental 530 -24186> 2013-04-03 10:33:48.266359 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e530 e530: 5 osds: 4 up, 5 in -24185> 2013-04-03 10:33:48.266380 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e530 update_from_paxos applying incremental 531 -24184> 2013-04-03 10:33:48.266410 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e531 e531: 5 osds: 5 up, 5 in -24183> 2013-04-03 10:33:48.266433 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e531 update_from_paxos applying incremental 532 -24182> 2013-04-03 10:33:48.266461 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e532 e532: 5 osds: 5 up, 5 in -24181> 2013-04-03 10:33:48.266484 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e532 update_from_paxos applying incremental 533 -24180> 2013-04-03 10:33:48.266512 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e533 e533: 5 osds: 5 up, 5 in -24179> 2013-04-03 10:33:48.266533 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e533 update_from_paxos applying incremental 534 -24178> 2013-04-03 10:33:48.266561 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e534 e534: 5 osds: 5 up, 5 in -24177> 2013-04-03 10:33:48.266589 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e534 update_from_paxos applying incremental 535 -24176> 2013-04-03 10:33:48.266618 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e535 e535: 5 osds: 4 up, 5 in -24175> 2013-04-03 10:33:48.266640 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e535 update_from_paxos applying incremental 536 -24174> 2013-04-03 10:33:48.266670 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e536 e536: 5 osds: 5 up, 5 in -24173> 2013-04-03 10:33:48.266693 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e536 update_from_paxos applying incremental 537 -24172> 2013-04-03 10:33:48.266721 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e537 e537: 5 osds: 5 up, 5 in -24171> 2013-04-03 10:33:48.266743 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e537 update_from_paxos applying incremental 538 -24170> 2013-04-03 10:33:48.266771 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e538 e538: 5 osds: 5 up, 5 in -24169> 2013-04-03 10:33:48.266793 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e538 update_from_paxos applying incremental 539 -24168> 2013-04-03 10:33:48.266826 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e539 e539: 5 osds: 5 up, 5 in -24167> 2013-04-03 10:33:48.266850 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e539 update_from_paxos applying incremental 540 -24166> 2013-04-03 10:33:48.266883 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e540 e540: 5 osds: 4 up, 5 in -24165> 2013-04-03 10:33:48.266921 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e540 update_from_paxos applying incremental 541 -24164> 2013-04-03 10:33:48.266956 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e541 e541: 5 osds: 5 up, 5 in -24163> 2013-04-03 10:33:48.266985 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e541 update_from_paxos applying incremental 542 -24162> 2013-04-03 10:33:48.267019 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e542 e542: 5 osds: 5 up, 5 in -24161> 2013-04-03 10:33:48.267046 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e542 update_from_paxos applying incremental 543 -24160> 2013-04-03 10:33:48.267078 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e543 e543: 5 osds: 5 up, 5 in -24159> 2013-04-03 10:33:48.267101 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e543 update_from_paxos applying incremental 544 -24158> 2013-04-03 10:33:48.267135 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e544 e544: 5 osds: 4 up, 5 in -24157> 2013-04-03 10:33:48.267157 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e544 update_from_paxos applying incremental 545 -24156> 2013-04-03 10:33:48.267191 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e545 e545: 5 osds: 5 up, 5 in -24155> 2013-04-03 10:33:48.267215 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e545 update_from_paxos applying incremental 546 -24154> 2013-04-03 10:33:48.267246 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e546 e546: 5 osds: 5 up, 5 in -24153> 2013-04-03 10:33:48.267269 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e546 update_from_paxos applying incremental 547 -24152> 2013-04-03 10:33:48.267300 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e547 e547: 5 osds: 4 up, 5 in -24151> 2013-04-03 10:33:48.267324 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e547 update_from_paxos applying incremental 548 -24150> 2013-04-03 10:33:48.267360 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e548 e548: 5 osds: 5 up, 5 in -24149> 2013-04-03 10:33:48.267389 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e548 update_from_paxos applying incremental 549 -24148> 2013-04-03 10:33:48.267423 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e549 e549: 5 osds: 5 up, 5 in -24147> 2013-04-03 10:33:48.267449 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e549 update_from_paxos applying incremental 550 -24146> 2013-04-03 10:33:48.267481 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e550 e550: 5 osds: 5 up, 5 in -24145> 2013-04-03 10:33:48.267504 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e550 update_from_paxos applying incremental 551 -24144> 2013-04-03 10:33:48.267536 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e551 e551: 5 osds: 4 up, 5 in -24143> 2013-04-03 10:33:48.267559 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e551 update_from_paxos applying incremental 552 -24142> 2013-04-03 10:33:48.267592 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e552 e552: 5 osds: 5 up, 5 in -24141> 2013-04-03 10:33:48.267616 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e552 update_from_paxos applying incremental 553 -24140> 2013-04-03 10:33:48.267648 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e553 e553: 5 osds: 5 up, 5 in -24139> 2013-04-03 10:33:48.267683 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e553 update_from_paxos applying incremental 554 -24138> 2013-04-03 10:33:48.267716 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e554 e554: 5 osds: 5 up, 5 in -24137> 2013-04-03 10:33:48.267744 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e554 update_from_paxos applying incremental 555 -24136> 2013-04-03 10:33:48.267775 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e555 e555: 5 osds: 4 up, 5 in -24135> 2013-04-03 10:33:48.267798 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e555 update_from_paxos applying incremental 556 -24134> 2013-04-03 10:33:48.267834 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e556 e556: 5 osds: 5 up, 5 in -24133> 2013-04-03 10:33:48.267857 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e556 update_from_paxos applying incremental 557 -24132> 2013-04-03 10:33:48.267891 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e557 e557: 5 osds: 5 up, 5 in -24131> 2013-04-03 10:33:48.267914 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e557 update_from_paxos applying incremental 558 -24130> 2013-04-03 10:33:48.267947 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e558 e558: 5 osds: 5 up, 5 in -24129> 2013-04-03 10:33:48.267970 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e558 update_from_paxos applying incremental 559 -24128> 2013-04-03 10:33:48.267998 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e559 e559: 5 osds: 4 up, 5 in -24127> 2013-04-03 10:33:48.268021 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e559 update_from_paxos applying incremental 560 -24126> 2013-04-03 10:33:48.268050 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e560 e560: 5 osds: 5 up, 5 in -24125> 2013-04-03 10:33:48.268072 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e560 update_from_paxos applying incremental 561 -24124> 2013-04-03 10:33:48.268100 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e561 e561: 5 osds: 5 up, 5 in -24123> 2013-04-03 10:33:48.268128 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e561 update_from_paxos applying incremental 562 -24122> 2013-04-03 10:33:48.268157 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e562 e562: 5 osds: 5 up, 5 in -24121> 2013-04-03 10:33:48.268180 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e562 update_from_paxos applying incremental 563 -24120> 2013-04-03 10:33:48.268208 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e563 e563: 5 osds: 4 up, 5 in -24119> 2013-04-03 10:33:48.268230 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e563 update_from_paxos applying incremental 564 -24118> 2013-04-03 10:33:48.268260 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e564 e564: 5 osds: 5 up, 5 in -24117> 2013-04-03 10:33:48.268282 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e564 update_from_paxos applying incremental 565 -24116> 2013-04-03 10:33:48.268310 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e565 e565: 5 osds: 5 up, 5 in -24115> 2013-04-03 10:33:48.268332 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e565 update_from_paxos applying incremental 566 -24114> 2013-04-03 10:33:48.268360 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e566 e566: 5 osds: 4 up, 5 in -24113> 2013-04-03 10:33:48.268382 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e566 update_from_paxos applying incremental 567 -24112> 2013-04-03 10:33:48.268412 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e567 e567: 5 osds: 5 up, 5 in -24111> 2013-04-03 10:33:48.268445 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e567 update_from_paxos applying incremental 568 -24110> 2013-04-03 10:33:48.268474 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e568 e568: 5 osds: 5 up, 5 in -24109> 2013-04-03 10:33:48.268496 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e568 update_from_paxos applying incremental 569 -24108> 2013-04-03 10:33:48.268524 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e569 e569: 5 osds: 5 up, 5 in -24107> 2013-04-03 10:33:48.268545 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e569 update_from_paxos applying incremental 570 -24106> 2013-04-03 10:33:48.268573 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e570 e570: 5 osds: 4 up, 5 in -24105> 2013-04-03 10:33:48.268594 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e570 update_from_paxos applying incremental 571 -24104> 2013-04-03 10:33:48.268623 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e571 e571: 5 osds: 5 up, 5 in -24103> 2013-04-03 10:33:48.268646 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e571 update_from_paxos applying incremental 572 -24102> 2013-04-03 10:33:48.268675 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e572 e572: 5 osds: 5 up, 5 in -24101> 2013-04-03 10:33:48.268698 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e572 update_from_paxos applying incremental 573 -24100> 2013-04-03 10:33:48.268727 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e573 e573: 5 osds: 5 up, 5 in -24099> 2013-04-03 10:33:48.268748 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e573 update_from_paxos applying incremental 574 -24098> 2013-04-03 10:33:48.268776 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e574 e574: 5 osds: 4 up, 5 in -24097> 2013-04-03 10:33:48.268799 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e574 update_from_paxos applying incremental 575 -24096> 2013-04-03 10:33:48.268829 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e575 e575: 5 osds: 3 up, 5 in -24095> 2013-04-03 10:33:48.268858 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e575 update_from_paxos applying incremental 576 -24094> 2013-04-03 10:33:48.268891 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e576 e576: 5 osds: 5 up, 5 in -24093> 2013-04-03 10:33:48.268915 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e576 update_from_paxos applying incremental 577 -24092> 2013-04-03 10:33:48.268948 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e577 e577: 5 osds: 5 up, 5 in -24091> 2013-04-03 10:33:48.268972 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e577 update_from_paxos applying incremental 578 -24090> 2013-04-03 10:33:48.269000 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e578 e578: 5 osds: 5 up, 5 in -24089> 2013-04-03 10:33:48.269036 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e578 update_from_paxos applying incremental 579 -24088> 2013-04-03 10:33:48.269066 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e579 e579: 5 osds: 5 up, 5 in -24087> 2013-04-03 10:33:48.269087 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e579 update_from_paxos applying incremental 580 -24086> 2013-04-03 10:33:48.269115 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e580 e580: 5 osds: 5 up, 5 in -24085> 2013-04-03 10:33:48.269137 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e580 update_from_paxos applying incremental 581 -24084> 2013-04-03 10:33:48.269165 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e581 e581: 5 osds: 4 up, 5 in -24083> 2013-04-03 10:33:48.269186 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e581 update_from_paxos applying incremental 582 -24082> 2013-04-03 10:33:48.269218 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e582 e582: 5 osds: 5 up, 5 in -24081> 2013-04-03 10:33:48.269241 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e582 update_from_paxos applying incremental 583 -24080> 2013-04-03 10:33:48.269269 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e583 e583: 5 osds: 5 up, 5 in -24079> 2013-04-03 10:33:48.269291 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e583 update_from_paxos applying incremental 584 -24078> 2013-04-03 10:33:48.269326 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e584 e584: 5 osds: 5 up, 5 in -24077> 2013-04-03 10:33:48.269348 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e584 update_from_paxos applying incremental 585 -24076> 2013-04-03 10:33:48.269376 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e585 e585: 5 osds: 4 up, 5 in -24075> 2013-04-03 10:33:48.269401 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e585 update_from_paxos applying incremental 586 -24074> 2013-04-03 10:33:48.269432 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e586 e586: 5 osds: 5 up, 5 in -24073> 2013-04-03 10:33:48.269556 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e586 update_from_paxos applying incremental 587 -24072> 2013-04-03 10:33:48.269593 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e587 e587: 5 osds: 5 up, 5 in -24071> 2013-04-03 10:33:48.269617 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e587 update_from_paxos applying incremental 588 -24070> 2013-04-03 10:33:48.269645 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e588 e588: 5 osds: 4 up, 5 in -24069> 2013-04-03 10:33:48.269665 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e588 update_from_paxos applying incremental 589 -24068> 2013-04-03 10:33:48.269695 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e589 e589: 5 osds: 5 up, 5 in -24067> 2013-04-03 10:33:48.269717 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e589 update_from_paxos applying incremental 590 -24066> 2013-04-03 10:33:48.269744 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e590 e590: 5 osds: 5 up, 5 in -24065> 2013-04-03 10:33:48.269766 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e590 update_from_paxos applying incremental 591 -24064> 2013-04-03 10:33:48.269794 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e591 e591: 5 osds: 5 up, 5 in -24063> 2013-04-03 10:33:48.269816 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e591 update_from_paxos applying incremental 592 -24062> 2013-04-03 10:33:48.269844 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e592 e592: 5 osds: 1 up, 5 in -24061> 2013-04-03 10:33:48.269866 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e592 update_from_paxos applying incremental 593 -24060> 2013-04-03 10:33:48.269900 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e593 e593: 5 osds: 5 up, 5 in -24059> 2013-04-03 10:33:48.269924 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e593 update_from_paxos applying incremental 594 -24058> 2013-04-03 10:33:48.269952 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e594 e594: 5 osds: 5 up, 5 in -24057> 2013-04-03 10:33:48.269974 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e594 update_from_paxos applying incremental 595 -24056> 2013-04-03 10:33:48.270002 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e595 e595: 5 osds: 5 up, 5 in -24055> 2013-04-03 10:33:48.270023 7fce0612e780 7 mon.hotcpc9039@-1(probing).osd e595 update_from_paxos applying incremental 596 -24054> 2013-04-03 10:33:48.270051 7fce0612e780 1 mon.hotcpc9039@-1(probing).osd e596 e596: 5 osds: 5 up, 5 in -24053> 2013-04-03 10:33:48.308522 7fce0612e780 10 mon.hotcpc9039@-1(probing).osd e596 send_to_waiting 596 -24052> 2013-04-03 10:33:48.308539 7fce0612e780 10 mon.hotcpc9039@-1(probing).osd e596 share_map_with_random_osd no up osd on our session map -24051> 2013-04-03 10:33:48.308542 7fce0612e780 10 mon.hotcpc9039@-1(probing).osd e596 update_logger -24050> 2013-04-03 10:33:48.308550 7fce0612e780 0 mon.hotcpc9039@-1(probing).osd e596 crush map has features 33816576, adjusting msgr requires -24049> 2013-04-03 10:33:48.308553 7fce0612e780 0 mon.hotcpc9039@-1(probing).osd e596 crush map has features 33816576, adjusting msgr requires -24048> 2013-04-03 10:33:48.308556 7fce0612e780 0 mon.hotcpc9039@-1(probing).osd e596 crush map has features 33816576, adjusting msgr requires -24047> 2013-04-03 10:33:48.308557 7fce0612e780 0 mon.hotcpc9039@-1(probing).osd e596 crush map has features 33816576, adjusting msgr requires -24046> 2013-04-03 10:33:48.308650 7fce0612e780 10 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos -24045> 2013-04-03 10:33:48.308694 7fce0612e780 10 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos version 393965 summary v 0 -24044> 2013-04-03 10:33:48.308711 7fce0612e780 10 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos latest full 393798 -24043> 2013-04-03 10:33:48.308743 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos loading summary e393798 -24042> 2013-04-03 10:33:48.308779 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos loaded summary e393798 -24041> 2013-04-03 10:33:48.308805 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393799 2013-04-03 10:20:51.337322 mon.0 10.252.24.25:6789/0 298 : [INF] pgmap v28783: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 2129B/s wr, 0op/s -24040> 2013-04-03 10:33:48.308841 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393800 2013-04-03 10:20:53.599023 mon.0 10.252.24.25:6789/0 299 : [INF] pgmap v28784: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 515B/s wr, 0op/s -24039> 2013-04-03 10:33:48.308867 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393801 2013-04-03 10:20:59.973381 mon.0 10.252.24.25:6789/0 300 : [INF] pgmap v28785: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1443B/s wr, 0op/s -24038> 2013-04-03 10:33:48.308891 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393802 2013-04-03 10:21:06.292502 mon.0 10.252.24.25:6789/0 301 : [INF] pgmap v28786: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1290B/s wr, 0op/s -24037> 2013-04-03 10:33:48.308916 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393803 2013-04-03 10:21:08.600498 mon.0 10.252.24.25:6789/0 302 : [INF] pgmap v28787: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -24036> 2013-04-03 10:33:48.308940 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393804 2013-04-03 10:21:14.820924 mon.0 10.252.24.25:6789/0 303 : [INF] pgmap v28788: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 2045B/s wr, 0op/s -24035> 2013-04-03 10:33:48.308965 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393805 2013-04-03 10:21:18.621050 mon.0 10.252.24.25:6789/0 304 : [INF] pgmap v28789: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1843B/s wr, 0op/s -24034> 2013-04-03 10:33:48.308990 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393806 2013-04-03 10:21:19.654331 mon.0 10.252.24.25:6789/0 305 : [INF] pgmap v28790: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 424B/s wr, 1op/s -24033> 2013-04-03 10:33:48.309015 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393807 2013-04-03 10:21:20.716665 mon.0 10.252.24.25:6789/0 306 : [INF] pgmap v28791: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1967B/s wr, 0op/s -24032> 2013-04-03 10:33:48.309039 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393808 2013-04-03 10:21:21.773809 mon.0 10.252.24.25:6789/0 307 : [INF] pgmap v28792: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 3873B/s wr, 0op/s -24031> 2013-04-03 10:33:48.309064 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393809 2013-04-03 10:21:23.831129 mon.0 10.252.24.25:6789/0 308 : [INF] pgmap v28793: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1951B/s wr, 2op/s -24030> 2013-04-03 10:33:48.309088 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393810 2013-04-03 10:21:26.297213 mon.0 10.252.24.25:6789/0 309 : [INF] pgmap v28794: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 1351B/s wr, 1op/s -24029> 2013-04-03 10:33:48.309113 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393811 2013-04-03 10:21:28.604666 mon.0 10.252.24.25:6789/0 310 : [INF] pgmap v28795: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1073B/s wr, 0op/s -24028> 2013-04-03 10:33:48.309138 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393812 2013-04-03 10:21:30.658033 mon.0 10.252.24.25:6789/0 311 : [INF] pgmap v28796: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s -24027> 2013-04-03 10:33:48.309166 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393813 2013-04-03 10:21:33.344174 mon.0 10.252.24.25:6789/0 312 : [INF] pgmap v28797: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1728B/s wr, 0op/s -24026> 2013-04-03 10:33:48.309190 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393814 2013-04-03 10:21:34.406515 mon.0 10.252.24.25:6789/0 313 : [INF] pgmap v28798: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail -24025> 2013-04-03 10:33:48.309215 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393815 2013-04-03 10:21:44.816675 mon.0 10.252.24.25:6789/0 314 : [INF] pgmap v28799: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 6787B/s wr, 0op/s -24024> 2013-04-03 10:33:48.309239 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393816 2013-04-03 10:21:51.295794 mon.0 10.252.24.25:6789/0 315 : [INF] pgmap v28800: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 4850B/s wr, 0op/s -24023> 2013-04-03 10:33:48.309264 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393817 2013-04-03 10:21:53.613451 mon.0 10.252.24.25:6789/0 316 : [INF] pgmap v28801: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -24022> 2013-04-03 10:33:48.309288 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393818 2013-04-03 10:21:59.830045 mon.0 10.252.24.25:6789/0 317 : [INF] pgmap v28802: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1563B/s wr, 0op/s -24021> 2013-04-03 10:33:48.309321 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393819 2013-04-03 10:22:06.296032 mon.0 10.252.24.25:6789/0 318 : [INF] pgmap v28803: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1290B/s wr, 0op/s -24020> 2013-04-03 10:33:48.309349 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393820 2013-04-03 10:22:14.812271 mon.0 10.252.24.25:6789/0 319 : [INF] pgmap v28804: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2184B/s wr, 0op/s -24019> 2013-04-03 10:33:48.309374 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393821 2013-04-03 10:22:21.299058 mon.0 10.252.24.25:6789/0 320 : [INF] pgmap v28805: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2184B/s wr, 0op/s -24018> 2013-04-03 10:33:48.309398 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393822 2013-04-03 10:22:23.619286 mon.0 10.252.24.25:6789/0 321 : [INF] pgmap v28806: 240 pgs: 240 active+clean; 252 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 582B/s wr, 0op/s -24017> 2013-04-03 10:33:48.309424 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393823 2013-04-03 10:22:26.304466 mon.0 10.252.24.25:6789/0 322 : [INF] pgmap v28807: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 409B/s wr, 0op/s -24016> 2013-04-03 10:33:48.309448 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393824 2013-04-03 10:22:29.852217 mon.0 10.252.24.25:6789/0 323 : [INF] pgmap v28808: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 12706B/s wr, 0op/s -24015> 2013-04-03 10:33:48.309473 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393825 2013-04-03 10:22:36.297701 mon.0 10.252.24.25:6789/0 324 : [INF] pgmap v28809: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 8191B/s wr, 0op/s -24014> 2013-04-03 10:33:48.309498 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393826 2013-04-03 10:22:39.819272 mon.0 10.252.24.25:6789/0 325 : [INF] pgmap v28810: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -24013> 2013-04-03 10:33:48.309527 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393827 2013-04-03 10:22:46.303710 mon.0 10.252.24.25:6789/0 326 : [INF] pgmap v28811: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -24012> 2013-04-03 10:33:48.309553 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393828 2013-04-03 10:22:48.614968 mon.0 10.252.24.25:6789/0 327 : [INF] pgmap v28812: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 582B/s wr, 0op/s -24011> 2013-04-03 10:33:48.309577 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393829 2013-04-03 10:22:59.835905 mon.0 10.252.24.25:6789/0 328 : [INF] pgmap v28813: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1288B/s wr, 0op/s -24010> 2013-04-03 10:33:48.309601 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393830 2013-04-03 10:23:06.304689 mon.0 10.252.24.25:6789/0 329 : [INF] pgmap v28814: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1157B/s wr, 0op/s -24009> 2013-04-03 10:33:48.309625 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393831 2013-04-03 10:23:08.617112 mon.0 10.252.24.25:6789/0 330 : [INF] pgmap v28815: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -24008> 2013-04-03 10:33:48.309649 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393832 2013-04-03 10:23:14.835365 mon.0 10.252.24.25:6789/0 331 : [INF] pgmap v28816: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2044B/s wr, 0op/s -24007> 2013-04-03 10:33:48.309673 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393833 2013-04-03 10:23:21.309477 mon.0 10.252.24.25:6789/0 332 : [INF] pgmap v28817: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1613B/s wr, 0op/s -24006> 2013-04-03 10:33:48.309697 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393834 2013-04-03 10:23:23.811733 mon.0 10.252.24.25:6789/0 333 : [INF] pgmap v28818: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 456B/s wr, 0op/s -24005> 2013-04-03 10:33:48.309722 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393835 2013-04-03 10:23:29.823881 mon.0 10.252.24.25:6789/0 334 : [INF] pgmap v28819: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1924B/s wr, 0op/s -24004> 2013-04-03 10:33:48.309746 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393836 2013-04-03 10:23:33.363780 mon.0 10.252.24.25:6789/0 335 : [INF] pgmap v28820: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1712B/s wr, 0op/s -24003> 2013-04-03 10:33:48.309770 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393837 2013-04-03 10:23:36.310693 mon.0 10.252.24.25:6789/0 336 : [INF] pgmap v28821: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 631B/s wr, 0op/s -24002> 2013-04-03 10:33:48.309795 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393838 2013-04-03 10:23:38.617406 mon.0 10.252.24.25:6789/0 337 : [INF] pgmap v28822: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 974B/s wr, 0op/s -24001> 2013-04-03 10:33:48.309821 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393839 2013-04-03 10:23:40.668033 mon.0 10.252.24.25:6789/0 338 : [INF] pgmap v28823: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3052B/s wr, 0op/s -24000> 2013-04-03 10:33:48.309846 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393840 2013-04-03 10:23:46.321817 mon.0 10.252.24.25:6789/0 339 : [INF] pgmap v28824: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2129B/s wr, 0op/s -23999> 2013-04-03 10:33:48.309874 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393841 2013-04-03 10:23:51.431045 mon.0 10.252.24.25:6789/0 340 : [INF] pgmap v28825: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 384B/s wr, 0op/s -23998> 2013-04-03 10:33:48.309899 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393842 2013-04-03 10:23:59.825837 mon.0 10.252.24.25:6789/0 341 : [INF] pgmap v28826: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 909B/s wr, 0op/s -23997> 2013-04-03 10:33:48.309923 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393843 2013-04-03 10:24:06.380496 mon.0 10.252.24.25:6789/0 342 : [INF] pgmap v28827: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -23996> 2013-04-03 10:33:48.309947 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393844 2013-04-03 10:24:08.621021 mon.0 10.252.24.25:6789/0 343 : [INF] pgmap v28828: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 582B/s wr, 0op/s -23995> 2013-04-03 10:33:48.309971 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393845 2013-04-03 10:24:10.680737 mon.0 10.252.24.25:6789/0 344 : [INF] pgmap v28829: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3046B/s wr, 0op/s -23994> 2013-04-03 10:33:48.309996 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393846 2013-04-03 10:24:16.329652 mon.0 10.252.24.25:6789/0 345 : [INF] pgmap v28830: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2129B/s wr, 0op/s -23993> 2013-04-03 10:33:48.310021 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393847 2013-04-03 10:24:24.831804 mon.0 10.252.24.25:6789/0 346 : [INF] pgmap v28831: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s -23992> 2013-04-03 10:33:48.310045 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393848 2013-04-03 10:24:28.620799 mon.0 10.252.24.25:6789/0 347 : [INF] pgmap v28832: 240 pgs: 240 active+clean; 252 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 15391B/s wr, 0op/s -23991> 2013-04-03 10:33:48.310070 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393849 2013-04-03 10:24:31.314710 mon.0 10.252.24.25:6789/0 348 : [INF] pgmap v28833: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 34569B/s wr, 1op/s -23990> 2013-04-03 10:33:48.310095 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393850 2013-04-03 10:24:33.640511 mon.0 10.252.24.25:6789/0 349 : [INF] pgmap v28834: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 9419B/s wr, 1op/s -23989> 2013-04-03 10:33:48.310120 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393851 2013-04-03 10:24:35.709497 mon.0 10.252.24.25:6789/0 350 : [INF] pgmap v28835: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 932B/s wr, 0op/s -23988> 2013-04-03 10:33:48.310144 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393852 2013-04-03 10:24:36.759875 mon.0 10.252.24.25:6789/0 351 : [INF] pgmap v28836: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3613B/s wr, 0op/s -23987> 2013-04-03 10:33:48.310169 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393853 2013-04-03 10:24:38.822593 mon.0 10.252.24.25:6789/0 352 : [INF] pgmap v28837: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2301B/s wr, 0op/s -23986> 2013-04-03 10:33:48.310194 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393854 2013-04-03 10:24:39.848998 mon.0 10.252.24.25:6789/0 353 : [INF] pgmap v28838: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3948B/s wr, 0op/s -23985> 2013-04-03 10:33:48.310210 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393854 2013-04-03 10:24:46.331439 mon.0 10.252.24.25:6789/0 354 : [INF] pgmap v28839: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2459B/s wr, 0op/s -23984> 2013-04-03 10:33:48.310241 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393855 2013-04-03 10:24:48.622594 mon.0 10.252.24.25:6789/0 355 : [INF] pgmap v28840: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 817B/s wr, 0op/s -23983> 2013-04-03 10:33:48.310267 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393856 2013-04-03 10:24:50.681984 mon.0 10.252.24.25:6789/0 356 : [INF] pgmap v28841: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1173B/s wr, 0op/s -23982> 2013-04-03 10:33:48.310293 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393857 2013-04-03 10:24:56.323478 mon.0 10.252.24.25:6789/0 357 : [INF] pgmap v28842: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1064B/s wr, 0op/s -23981> 2013-04-03 10:33:48.310317 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393858 2013-04-03 10:25:04.834558 mon.0 10.252.24.25:6789/0 358 : [INF] pgmap v28843: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1157B/s wr, 0op/s -23980> 2013-04-03 10:33:48.310341 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393859 2013-04-03 10:25:09.843524 mon.0 10.252.24.25:6789/0 359 : [INF] pgmap v28844: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 54244B/s wr, 0op/s -23979> 2013-04-03 10:33:48.310367 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393860 2013-04-03 10:25:11.945807 mon.0 10.252.24.25:6789/0 360 : [INF] pgmap v28845: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 100KB/s wr, 0op/s -23978> 2013-04-03 10:33:48.310391 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393861 2013-04-03 10:25:14.141112 mon.0 10.252.24.25:6789/0 361 : [INF] pgmap v28846: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1194B/s wr, 0op/s -23977> 2013-04-03 10:33:48.310415 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393862 2013-04-03 10:25:16.319726 mon.0 10.252.24.25:6789/0 362 : [INF] pgmap v28847: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1159B/s wr, 0op/s -23976> 2013-04-03 10:33:48.310440 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393863 2013-04-03 10:25:23.806103 mon.0 10.252.24.25:6789/0 363 : [INF] pgmap v28848: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 422B/s wr, 0op/s -23975> 2013-04-03 10:33:48.310464 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393864 2013-04-03 10:25:24.870193 mon.0 10.252.24.25:6789/0 364 : [INF] pgmap v28849: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 962B/s wr, 0op/s -23974> 2013-04-03 10:33:48.310489 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393865 2013-04-03 10:25:31.321772 mon.0 10.252.24.25:6789/0 365 : [INF] pgmap v28850: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1634B/s wr, 0op/s -23973> 2013-04-03 10:33:48.310513 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393866 2013-04-03 10:25:33.377553 mon.0 10.252.24.25:6789/0 366 : [INF] pgmap v28851: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 479B/s wr, 0op/s -23972> 2013-04-03 10:33:48.310538 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393867 2013-04-03 10:25:34.440972 mon.0 10.252.24.25:6789/0 367 : [INF] pgmap v28852: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 331B/s wr, 0op/s -23971> 2013-04-03 10:33:48.310563 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393868 2013-04-03 10:25:35.611234 mon.0 10.252.24.25:6789/0 368 : [INF] pgmap v28853: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 4409B/s wr, 0op/s -23970> 2013-04-03 10:33:48.310580 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393868 2013-04-03 10:25:41.321511 mon.0 10.252.24.25:6789/0 369 : [INF] pgmap v28854: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1776B/s wr, 0op/s -23969> 2013-04-03 10:33:48.310610 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393869 2013-04-03 10:25:49.855738 mon.0 10.252.24.25:6789/0 370 : [INF] pgmap v28855: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1425B/s wr, 0op/s -23968> 2013-04-03 10:33:48.310635 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393870 2013-04-03 10:25:56.324958 mon.0 10.252.24.25:6789/0 371 : [INF] pgmap v28856: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -23967> 2013-04-03 10:33:48.310659 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393871 2013-04-03 10:25:58.632306 mon.0 10.252.24.25:6789/0 372 : [INF] pgmap v28857: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 582B/s wr, 0op/s -23966> 2013-04-03 10:33:48.310684 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393872 2013-04-03 10:26:00.047557 mon.0 10.252.24.25:6789/0 373 : [INF] pgmap v28858: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3786B/s wr, 0op/s -23965> 2013-04-03 10:33:48.310708 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393873 2013-04-03 10:26:06.325083 mon.0 10.252.24.25:6789/0 374 : [INF] pgmap v28859: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2129B/s wr, 0op/s -23964> 2013-04-03 10:33:48.310733 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393874 2013-04-03 10:26:14.846182 mon.0 10.252.24.25:6789/0 375 : [INF] pgmap v28860: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -23963> 2013-04-03 10:33:48.310759 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393875 2013-04-03 10:26:21.323277 mon.0 10.252.24.25:6789/0 376 : [INF] pgmap v28861: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -23962> 2013-04-03 10:33:48.310783 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393876 2013-04-03 10:26:23.634318 mon.0 10.252.24.25:6789/0 377 : [INF] pgmap v28862: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 582B/s wr, 0op/s -23961> 2013-04-03 10:33:48.310807 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393877 2013-04-03 10:26:29.858498 mon.0 10.252.24.25:6789/0 378 : [INF] pgmap v28863: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1563B/s wr, 0op/s -23960> 2013-04-03 10:33:48.310832 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393878 2013-04-03 10:26:36.325531 mon.0 10.252.24.25:6789/0 379 : [INF] pgmap v28864: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1290B/s wr, 0op/s -23959> 2013-04-03 10:33:48.310856 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393879 2013-04-03 10:26:59.868376 mon.0 10.252.24.25:6789/0 380 : [INF] pgmap v28865: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3276B/s wr, 0op/s -23958> 2013-04-03 10:33:48.310881 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393880 2013-04-03 10:27:06.589212 mon.0 10.252.24.25:6789/0 381 : [INF] pgmap v28866: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3276B/s wr, 0op/s -23957> 2013-04-03 10:33:48.310906 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393881 2013-04-03 10:27:08.654340 mon.0 10.252.24.25:6789/0 382 : [INF] pgmap v28867: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 581B/s wr, 0op/s -23956> 2013-04-03 10:33:48.310931 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393882 2013-04-03 10:27:14.881423 mon.0 10.252.24.25:6789/0 383 : [INF] pgmap v28868: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3005B/s wr, 0op/s -23955> 2013-04-03 10:33:48.310958 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393883 2013-04-03 10:27:21.335498 mon.0 10.252.24.25:6789/0 384 : [INF] pgmap v28869: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2261B/s wr, 0op/s -23954> 2013-04-03 10:33:48.310984 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393884 2013-04-03 10:27:23.833427 mon.0 10.252.24.25:6789/0 385 : [INF] pgmap v28870: 240 pgs: 240 active+clean; 253 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 456B/s wr, 0op/s -23953> 2013-04-03 10:33:48.311008 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393885 2013-04-03 10:27:29.859601 mon.0 10.252.24.25:6789/0 386 : [INF] pgmap v28871: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 8656B/s wr, 0op/s -23952> 2013-04-03 10:33:48.311032 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393886 2013-04-03 10:27:33.411017 mon.0 10.252.24.25:6789/0 387 : [INF] pgmap v28872: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 7689B/s wr, 0op/s -23951> 2013-04-03 10:33:48.311056 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393887 2013-04-03 10:27:36.335155 mon.0 10.252.24.25:6789/0 388 : [INF] pgmap v28873: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 631B/s wr, 0op/s -23950> 2013-04-03 10:33:48.311080 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393888 2013-04-03 10:27:38.644414 mon.0 10.252.24.25:6789/0 389 : [INF] pgmap v28874: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 977B/s wr, 0op/s -23949> 2013-04-03 10:33:48.311104 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393889 2013-04-03 10:27:44.871932 mon.0 10.252.24.25:6789/0 390 : [INF] pgmap v28875: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2043B/s wr, 0op/s -23948> 2013-04-03 10:33:48.311128 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393890 2013-04-03 10:27:51.334646 mon.0 10.252.24.25:6789/0 391 : [INF] pgmap v28876: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1613B/s wr, 0op/s -23947> 2013-04-03 10:33:48.311153 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393891 2013-04-03 10:27:59.874893 mon.0 10.252.24.25:6789/0 392 : [INF] pgmap v28877: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -23946> 2013-04-03 10:33:48.311178 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393892 2013-04-03 10:28:06.339712 mon.0 10.252.24.25:6789/0 393 : [INF] pgmap v28878: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -23945> 2013-04-03 10:33:48.311203 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393893 2013-04-03 10:28:08.642113 mon.0 10.252.24.25:6789/0 394 : [INF] pgmap v28879: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 582B/s wr, 0op/s -23944> 2013-04-03 10:33:48.311228 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393894 2013-04-03 10:28:15.019587 mon.0 10.252.24.25:6789/0 395 : [INF] pgmap v28880: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1562B/s wr, 0op/s -23943> 2013-04-03 10:33:48.311253 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393895 2013-04-03 10:28:21.366991 mon.0 10.252.24.25:6789/0 396 : [INF] pgmap v28881: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1290B/s wr, 0op/s -23942> 2013-04-03 10:33:48.311278 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393896 2013-04-03 10:28:24.854986 mon.0 10.252.24.25:6789/0 397 : [INF] pgmap v28882: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -23941> 2013-04-03 10:33:48.311305 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393897 2013-04-03 10:28:31.338804 mon.0 10.252.24.25:6789/0 398 : [INF] pgmap v28883: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -23940> 2013-04-03 10:33:48.311330 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393898 2013-04-03 10:28:33.648626 mon.0 10.252.24.25:6789/0 399 : [INF] pgmap v28884: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 582B/s wr, 0op/s -23939> 2013-04-03 10:33:48.311354 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393899 2013-04-03 10:28:39.980720 mon.0 10.252.24.25:6789/0 400 : [INF] pgmap v28885: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1562B/s wr, 0op/s -23938> 2013-04-03 10:33:48.311377 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393900 2013-04-03 10:28:51.689744 mon.0 10.252.24.25:6789/0 401 : [INF] pgmap v28886: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 920B/s wr, 0op/s -23937> 2013-04-03 10:33:48.311401 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393901 2013-04-03 10:28:54.866292 mon.0 10.252.24.25:6789/0 402 : [INF] pgmap v28887: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -23936> 2013-04-03 10:33:48.311425 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393902 2013-04-03 10:29:01.619852 mon.0 10.252.24.25:6789/0 403 : [INF] pgmap v28888: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1654B/s wr, 0op/s -23935> 2013-04-03 10:33:48.311450 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393903 2013-04-03 10:29:03.880345 mon.0 10.252.24.25:6789/0 404 : [INF] pgmap v28889: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 571B/s wr, 0op/s -23934> 2013-04-03 10:33:48.311475 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393904 2013-04-03 10:29:05.302835 mon.0 10.252.24.25:6789/0 405 : [INF] pgmap v28890: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2584B/s wr, 0op/s -23933> 2013-04-03 10:33:48.311499 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393905 2013-04-03 10:29:14.939829 mon.0 10.252.24.25:6789/0 406 : [INF] pgmap v28891: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1484B/s wr, 0op/s -23932> 2013-04-03 10:33:48.311523 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393906 2013-04-03 10:29:18.343941 mon.0 10.252.24.25:6789/0 407 : [INF] pgmap v28892: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 915B/s wr, 0op/s -23931> 2013-04-03 10:33:48.311547 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393907 2013-04-03 10:29:20.714172 mon.0 10.252.24.25:6789/0 408 : [INF] pgmap v28893: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1462B/s wr, 0op/s -23930> 2013-04-03 10:33:48.311572 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393908 2013-04-03 10:29:21.752397 mon.0 10.252.24.25:6789/0 409 : [INF] pgmap v28894: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2403B/s wr, 0op/s -23929> 2013-04-03 10:33:48.311588 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393908 2013-04-03 10:29:23.806558 mon.0 10.252.24.25:6789/0 410 : [INF] pgmap v28895: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1551B/s wr, 0op/s -23928> 2013-04-03 10:33:48.311613 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393909 2013-04-03 10:29:24.931887 mon.0 10.252.24.25:6789/0 411 : [INF] pgmap v28896: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 331B/s wr, 0op/s -23927> 2013-04-03 10:33:48.311630 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393909 2013-04-03 10:29:26.437566 mon.0 10.252.24.25:6789/0 412 : [INF] pgmap v28897: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1586B/s wr, 0op/s -23926> 2013-04-03 10:33:48.311658 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393910 2013-04-03 10:29:28.849602 mon.0 10.252.24.25:6789/0 413 : [INF] pgmap v28898: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1041B/s wr, 0op/s -23925> 2013-04-03 10:33:48.311684 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393911 2013-04-03 10:29:29.920118 mon.0 10.252.24.25:6789/0 414 : [INF] pgmap v28899: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail -23924> 2013-04-03 10:33:48.311702 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393911 2013-04-03 10:29:31.414509 mon.0 10.252.24.25:6789/0 415 : [INF] pgmap v28900: 240 pgs: 240 active+clean; 257 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1233KB/s wr, 5op/s -23923> 2013-04-03 10:33:48.311727 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393912 2013-04-03 10:29:33.644207 mon.0 10.252.24.25:6789/0 416 : [INF] pgmap v28901: 240 pgs: 240 active+clean; 257 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 850KB/s wr, 3op/s -23922> 2013-04-03 10:33:48.311751 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393913 2013-04-03 10:29:34.675722 mon.0 10.252.24.25:6789/0 417 : [INF] pgmap v28902: 240 pgs: 240 active+clean; 257 MB data, 5661 MB used, 1067 GB / 1079 GB avail -23921> 2013-04-03 10:33:48.311768 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393913 2013-04-03 10:29:35.732838 mon.0 10.252.24.25:6789/0 418 : [INF] pgmap v28903: 240 pgs: 240 active+clean; 257 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 5833B/s wr, 0op/s -23920> 2013-04-03 10:33:48.311793 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393914 2013-04-03 10:29:36.831945 mon.0 10.252.24.25:6789/0 419 : [INF] pgmap v28904: 240 pgs: 240 active+clean; 257 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 13732B/s wr, 2op/s -23919> 2013-04-03 10:33:48.311817 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393915 2013-04-03 10:29:41.350938 mon.0 10.252.24.25:6789/0 420 : [INF] pgmap v28905: 240 pgs: 240 active+clean; 257 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 3826B/s wr, 1op/s -23918> 2013-04-03 10:33:48.311841 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393916 2013-04-03 10:29:43.781683 mon.0 10.252.24.25:6789/0 421 : [INF] pgmap v28906: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 876B/s wr, 0op/s -23917> 2013-04-03 10:33:48.311866 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393917 2013-04-03 10:29:46.357635 mon.0 10.252.24.25:6789/0 422 : [INF] pgmap v28907: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 1433B/s wr, 0op/s -23916> 2013-04-03 10:33:48.311890 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393918 2013-04-03 10:29:49.875163 mon.0 10.252.24.25:6789/0 423 : [INF] pgmap v28908: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3692B/s wr, 0op/s -23915> 2013-04-03 10:33:48.311914 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393919 2013-04-03 10:29:52.149051 mon.0 10.252.24.25:6789/0 424 : [INF] pgmap v28909: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3222B/s wr, 0op/s -23914> 2013-04-03 10:33:48.311938 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393920 2013-04-03 10:29:56.352544 mon.0 10.252.24.25:6789/0 425 : [INF] pgmap v28910: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 947B/s wr, 0op/s -23913> 2013-04-03 10:33:48.311964 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393921 2013-04-03 10:30:00.028485 mon.0 10.252.24.25:6789/0 426 : [INF] pgmap v28911: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 2100B/s wr, 0op/s -23912> 2013-04-03 10:33:48.311991 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393922 2013-04-03 10:30:06.353747 mon.0 10.252.24.25:6789/0 427 : [INF] pgmap v28912: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -23911> 2013-04-03 10:33:48.312016 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393923 2013-04-03 10:30:08.868481 mon.0 10.252.24.25:6789/0 428 : [INF] pgmap v28913: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 575B/s wr, 0op/s -23910> 2013-04-03 10:33:48.312040 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393924 2013-04-03 10:30:15.024813 mon.0 10.252.24.25:6789/0 429 : [INF] pgmap v28914: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 84726B/s wr, 0op/s -23909> 2013-04-03 10:33:48.312064 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393925 2013-04-03 10:30:21.351772 mon.0 10.252.24.25:6789/0 430 : [INF] pgmap v28915: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 57620B/s wr, 0op/s -23908> 2013-04-03 10:33:48.312088 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393926 2013-04-03 10:30:26.363841 mon.0 10.252.24.25:6789/0 431 : [INF] pgmap v28916: 240 pgs: 240 active+clean; 257 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 356B/s wr, 0op/s -23907> 2013-04-03 10:33:48.312112 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393927 2013-04-03 10:30:29.961084 mon.0 10.252.24.25:6789/0 432 : [INF] pgmap v28917: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1442B/s wr, 0op/s -23906> 2013-04-03 10:33:48.312138 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393928 2013-04-03 10:30:36.359354 mon.0 10.252.24.25:6789/0 433 : [INF] pgmap v28918: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -23905> 2013-04-03 10:33:48.312162 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393929 2013-04-03 10:30:38.936804 mon.0 10.252.24.25:6789/0 434 : [INF] pgmap v28919: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 575B/s wr, 0op/s -23904> 2013-04-03 10:33:48.312187 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393930 2013-04-03 10:30:44.897401 mon.0 10.252.24.25:6789/0 435 : [INF] pgmap v28920: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 2043B/s wr, 0op/s -23903> 2013-04-03 10:33:48.312211 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393931 2013-04-03 10:30:51.371081 mon.0 10.252.24.25:6789/0 436 : [INF] pgmap v28921: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1627B/s wr, 0op/s -23902> 2013-04-03 10:33:48.312237 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393932 2013-04-03 10:30:59.897645 mon.0 10.252.24.25:6789/0 437 : [INF] pgmap v28922: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -23901> 2013-04-03 10:33:48.312262 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393933 2013-04-03 10:31:06.360510 mon.0 10.252.24.25:6789/0 438 : [INF] pgmap v28923: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -23900> 2013-04-03 10:33:48.312287 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393934 2013-04-03 10:31:08.779599 mon.0 10.252.24.25:6789/0 439 : [INF] pgmap v28924: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 575B/s wr, 0op/s -23899> 2013-04-03 10:33:48.312312 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393935 2013-04-03 10:31:14.917091 mon.0 10.252.24.25:6789/0 440 : [INF] pgmap v28925: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1562B/s wr, 0op/s -23898> 2013-04-03 10:33:48.312339 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393936 2013-04-03 10:31:21.362760 mon.0 10.252.24.25:6789/0 441 : [INF] pgmap v28926: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1302B/s wr, 0op/s -23897> 2013-04-03 10:33:48.312364 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393937 2013-04-03 10:31:24.887390 mon.0 10.252.24.25:6789/0 442 : [INF] pgmap v28927: 240 pgs: 240 active+clean; 257 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -23896> 2013-04-03 10:33:48.312388 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393938 2013-04-03 10:31:28.860900 mon.0 10.252.24.25:6789/0 443 : [INF] pgmap v28928: 240 pgs: 240 active+clean; 257 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1642B/s wr, 0op/s -23895> 2013-04-03 10:33:48.312412 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393939 2013-04-03 10:31:31.364892 mon.0 10.252.24.25:6789/0 444 : [INF] pgmap v28929: 240 pgs: 240 active+clean; 257 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 631B/s wr, 0op/s -23894> 2013-04-03 10:33:48.312436 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393940 2013-04-03 10:31:33.994861 mon.0 10.252.24.25:6789/0 445 : [INF] pgmap v28930: 240 pgs: 240 active+clean; 257 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 872B/s wr, 0op/s -23893> 2013-04-03 10:33:48.312460 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393941 2013-04-03 10:31:35.073774 mon.0 10.252.24.25:6789/0 446 : [INF] pgmap v28931: 240 pgs: 240 active+clean; 257 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 279B/s wr, 0op/s -23892> 2013-04-03 10:33:48.312484 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393942 2013-04-03 10:31:44.899918 mon.0 10.252.24.25:6789/0 447 : [INF] pgmap v28932: 240 pgs: 240 active+clean; 257 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1534B/s wr, 0op/s -23891> 2013-04-03 10:33:48.312510 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393943 2013-04-03 10:31:51.369106 mon.0 10.252.24.25:6789/0 448 : [INF] pgmap v28933: 240 pgs: 240 active+clean; 257 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1253B/s wr, 0op/s -23890> 2013-04-03 10:33:48.312534 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393944 2013-04-03 10:31:53.787905 mon.0 10.252.24.25:6789/0 449 : [INF] pgmap v28934: 240 pgs: 240 active+clean; 257 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 575B/s wr, 0op/s -23889> 2013-04-03 10:33:48.312559 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393945 2013-04-03 10:31:59.893972 mon.0 10.252.24.25:6789/0 450 : [INF] pgmap v28935: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9253B/s wr, 0op/s -23888> 2013-04-03 10:33:48.312583 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393946 2013-04-03 10:32:06.367714 mon.0 10.252.24.25:6789/0 451 : [INF] pgmap v28936: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 6510B/s wr, 0op/s -23887> 2013-04-03 10:33:48.312609 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393947 2013-04-03 10:32:09.940177 mon.0 10.252.24.25:6789/0 452 : [INF] pgmap v28937: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2866B/s wr, 0op/s -23886> 2013-04-03 10:33:48.312634 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393948 2013-04-03 10:32:16.364106 mon.0 10.252.24.25:6789/0 453 : [INF] pgmap v28938: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2866B/s wr, 0op/s -23885> 2013-04-03 10:33:48.312659 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393949 2013-04-03 10:32:18.807721 mon.0 10.252.24.25:6789/0 454 : [INF] pgmap v28939: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 575B/s wr, 0op/s -23884> 2013-04-03 10:33:48.312686 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393950 2013-04-03 10:32:34.892937 mon.0 10.252.24.25:6789/0 455 : [INF] pgmap v28940: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4256B/s wr, 0op/s -23883> 2013-04-03 10:33:48.312711 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393951 2013-04-03 10:32:41.384322 mon.0 10.252.24.25:6789/0 456 : [INF] pgmap v28941: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 3627B/s wr, 0op/s -23882> 2013-04-03 10:33:48.312735 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393952 2013-04-03 10:32:43.851719 mon.0 10.252.24.25:6789/0 457 : [INF] pgmap v28942: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 575B/s wr, 0op/s -23881> 2013-04-03 10:33:48.312759 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393953 2013-04-03 10:32:44.925569 mon.0 10.252.24.25:6789/0 458 : [INF] pgmap v28943: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 3792B/s wr, 0op/s -23880> 2013-04-03 10:33:48.312784 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393954 2013-04-03 10:32:51.410017 mon.0 10.252.24.25:6789/0 459 : [INF] pgmap v28944: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2160B/s wr, 0op/s -23879> 2013-04-03 10:33:48.312808 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393955 2013-04-03 10:32:54.935886 mon.0 10.252.24.25:6789/0 460 : [INF] pgmap v28945: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2454B/s wr, 0op/s -23878> 2013-04-03 10:33:48.312832 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393956 2013-04-03 10:33:01.486728 mon.0 10.252.24.25:6789/0 461 : [INF] pgmap v28946: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2457B/s wr, 0op/s -23877> 2013-04-03 10:33:48.312857 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393957 2013-04-03 10:33:03.931473 mon.0 10.252.24.25:6789/0 462 : [INF] pgmap v28947: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 574B/s wr, 0op/s -23876> 2013-04-03 10:33:48.312882 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393958 2013-04-03 10:33:14.919978 mon.0 10.252.24.25:6789/0 463 : [INF] pgmap v28948: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1287B/s wr, 0op/s -23875> 2013-04-03 10:33:48.312906 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393959 2013-04-03 10:33:21.404793 mon.0 10.252.24.25:6789/0 464 : [INF] pgmap v28949: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1223B/s wr, 0op/s -23874> 2013-04-03 10:33:48.312931 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393960 2013-04-03 10:33:24.055580 mon.0 10.252.24.25:6789/0 465 : [INF] pgmap v28950: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 690B/s wr, 0op/s -23873> 2013-04-03 10:33:48.312955 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393961 2013-04-03 10:33:28.964123 mon.0 10.252.24.25:6789/0 466 : [INF] pgmap v28951: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 136B/s wr, 0op/s -23872> 2013-04-03 10:33:48.312980 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393962 2013-04-03 10:33:30.015496 mon.0 10.252.24.25:6789/0 467 : [INF] pgmap v28952: 240 pgs: 240 active+clean; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1982B/s wr, 0op/s -23871> 2013-04-03 10:33:48.312997 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393962 2013-04-03 10:33:33.475513 mon.0 10.252.24.25:6789/0 468 : [INF] pgmap v28953: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 2661B/s wr, 0op/s -23870> 2013-04-03 10:33:48.313021 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393963 2013-04-03 10:33:36.381925 mon.0 10.252.24.25:6789/0 469 : [INF] pgmap v28954: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 641B/s wr, 0op/s -23869> 2013-04-03 10:33:48.313046 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393964 2013-04-03 10:33:42.787775 mon.0 10.252.24.25:6789/0 470 : [INF] osdmap e596: 5 osds: 5 up, 5 in -23868> 2013-04-03 10:33:48.313063 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393964 2013-04-03 10:33:42.867348 mon.0 10.252.24.25:6789/0 471 : [INF] mds.? 10.252.24.25:6802/3144 up:boot -23867> 2013-04-03 10:33:48.313082 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393964 2013-04-03 10:33:42.867743 mon.0 10.252.24.25:6789/0 472 : [INF] mdsmap e183: 0/1/1 up, 3 up:standby, 1 failed -23866> 2013-04-03 10:33:48.313100 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393964 2013-04-03 10:33:42.964731 mon.0 10.252.24.25:6789/0 473 : [INF] pgmap v28955: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 438B/s wr, 0op/s -23865> 2013-04-03 10:33:48.313125 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393965 2013-04-03 10:33:42.990674 mon.0 10.252.24.25:6789/0 474 : [INF] mdsmap e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby -23864> 2013-04-03 10:33:48.313142 7fce0612e780 7 mon.hotcpc9039@-1(probing).log v393965 update_from_paxos applying incremental log 393965 2013-04-03 10:33:44.310596 mon.0 10.252.24.25:6789/0 475 : [INF] pgmap v28956: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail -23863> 2013-04-03 10:33:48.313221 7fce0612e780 10 mon.hotcpc9039@-1(probing).log v393965 check_subs -23862> 2013-04-03 10:33:48.313262 7fce0612e780 10 mon.hotcpc9039@-1(probing).monmap v3 update_from_paxos version 5, my v 3 -23861> 2013-04-03 10:33:48.313274 7fce0612e780 10 mon.hotcpc9039@-1(probing).monmap v3 update_from_paxos got 5 -23860> 2013-04-03 10:33:48.313292 7fce0612e780 0 mon.hotcpc9039@-1(probing).paxos(paxos recovering c 427161..427188) prepare_bootstrap -23859> 2013-04-03 10:33:48.313300 7fce0612e780 10 mon.hotcpc9039@-1(probing).auth v2059 update_from_paxos -23858> 2013-04-03 10:33:48.313659 7fce0612e780 10 mon.hotcpc9039@-1(probing).auth v2059 update_from_paxos version 2059 keys ver 0 latest 2039 -23857> 2013-04-03 10:33:48.313681 7fce0612e780 7 mon.hotcpc9039@-1(probing).auth v2059 update_from_paxos loading summary e 2039 -23856> 2013-04-03 10:33:48.313690 7fce0612e780 7 mon.hotcpc9039@-1(probing).auth v2059 update_from_paxos latest length 1901 -23855> 2013-04-03 10:33:48.313740 7fce0612e780 10 mon.hotcpc9039@-1(probing).auth v2059 update_from_paxos key server version 2039 -23854> 2013-04-03 10:33:48.314361 7fce0612e780 10 mon.hotcpc9039@-1(probing).auth v2059 update_from_paxos() last_allocated_id=11896 max_global_id=11896 -23853> 2013-04-03 10:33:48.314481 7fce0612e780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mon/cit_nonfood-hotcpc9039/keyring -23852> 2013-04-03 10:33:48.314488 7fce0612e780 5 asok(0x1b9d000) register_command mon_status hook 0x1b91440 -23851> 2013-04-03 10:33:48.314492 7fce0612e780 5 asok(0x1b9d000) register_command quorum_status hook 0x1b91440 -23850> 2013-04-03 10:33:48.314495 7fce0612e780 5 asok(0x1b9d000) register_command sync_status hook 0x1b91440 -23849> 2013-04-03 10:33:48.314497 7fce0612e780 5 asok(0x1b9d000) register_command add_bootstrap_peer_hint hook 0x1b91440 -23848> 2013-04-03 10:33:48.315103 7fce0612e780 1 finished global_init_daemonize -23847> 2013-04-03 10:33:48.317964 7fce0612e780 5 asok(0x1b9d000) init /var/run/ceph/ceph-mon.hotcpc9039.asok -23846> 2013-04-03 10:33:48.317981 7fce0612e780 5 asok(0x1b9d000) bind_and_listen /var/run/ceph/ceph-mon.hotcpc9039.asok -23845> 2013-04-03 10:33:48.318011 7fce0612e780 5 asok(0x1b9d000) register_command 0 hook 0x1b90d40 -23844> 2013-04-03 10:33:48.318018 7fce0612e780 5 asok(0x1b9d000) register_command version hook 0x1b90d40 -23843> 2013-04-03 10:33:48.318026 7fce0612e780 5 asok(0x1b9d000) register_command git_version hook 0x1b90d40 -23842> 2013-04-03 10:33:48.318029 7fce0612e780 5 asok(0x1b9d000) register_command help hook 0x1b913a0 -23841> 2013-04-03 10:33:48.318053 7fce01cdf700 5 asok(0x1b9d000) entry start -23840> 2013-04-03 10:33:48.318090 7fce0612e780 1 -- 10.252.24.25:6789/0 messenger.start -23839> 2013-04-03 10:33:48.318121 7fce0612e780 2 mon.hotcpc9039@-1(probing) e5 init -23838> 2013-04-03 10:33:48.318170 7fce0612e780 1 accepter.accepter.start -23837> 2013-04-03 10:33:48.318189 7fce0612e780 10 mon.hotcpc9039@-1(probing) e5 bootstrap -23836> 2013-04-03 10:33:48.318195 7fce0612e780 10 mon.hotcpc9039@-1(probing) e5 unregister_cluster_logger - not registered -23835> 2013-04-03 10:33:48.318197 7fce0612e780 10 mon.hotcpc9039@-1(probing) e5 cancel_probe_timeout (none scheduled) -23834> 2013-04-03 10:33:48.318199 7fce0612e780 0 mon.hotcpc9039@-1(probing) e5 my rank is now 0 (was -1) -23833> 2013-04-03 10:33:48.318202 7fce0612e780 1 -- 10.252.24.25:6789/0 mark_down_all -23832> 2013-04-03 10:33:48.318207 7fce0612e780 10 mon.hotcpc9039@0(probing) e5 reset_sync -23831> 2013-04-03 10:33:48.318210 7fce0612e780 10 mon.hotcpc9039@0(probing) e5 reset -23830> 2013-04-03 10:33:48.318212 7fce0612e780 10 mon.hotcpc9039@0(probing) e5 timecheck_finish -23829> 2013-04-03 10:33:48.318227 7fce0612e780 10 mon.hotcpc9039@0(probing) e5 cancel_probe_timeout (none scheduled) -23828> 2013-04-03 10:33:48.318230 7fce0612e780 10 mon.hotcpc9039@0(probing) e5 reset_probe_timeout 0x1b91350 after 2 seconds -23827> 2013-04-03 10:33:48.318258 7fce0612e780 10 mon.hotcpc9039@0(probing) e5 probing other monitors -23826> 2013-04-03 10:33:48.318264 7fce0612e780 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039) v4 -- ?+0 0x1fbd900 -23825> 2013-04-03 10:33:48.318309 7fce0612e780 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039) v4 -- ?+0 0x1fbd600 -23824> 2013-04-03 10:33:48.318864 7fce027bf700 10 mon.hotcpc9039@0(probing) e5 ms_get_authorizer for mon -23823> 2013-04-03 10:33:48.319453 7fce027bf700 10 mon.hotcpc9039@0(probing) e5 ms_get_authorizer for mon -23822> 2013-04-03 10:33:48.320773 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112833 ==== mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc12533 quorum 0,2 paxos( fc 427161 lc 427188 )) v4 ==== 601+0+0 (2908846 0 0) 0x1fbd600 con 0x208a840 -23821> 2013-04-03 10:33:48.320796 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 do not have session, making new one -23820> 2013-04-03 10:33:48.320804 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 ms_dispatch new session MonSession: mon.2 10.252.24.88:6789/0 is open for mon.2 10.252.24.88:6789/0 -23819> 2013-04-03 10:33:48.320811 7fcdffcdb700 5 mon.hotcpc9039@0(probing) e5 setting monitor caps on this connection -23818> 2013-04-03 10:33:48.320814 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 handle_probe mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc12533 quorum 0,2 paxos( fc 427161 lc 427188 )) v4 -23817> 2013-04-03 10:33:48.320819 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 handle_probe_reply mon.2 10.252.24.88:6789/0mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc12533 quorum 0,2 paxos( fc 427161 lc 427188 )) v4 -23816> 2013-04-03 10:33:48.320827 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 monmap is e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -23815> 2013-04-03 10:33:48.320840 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 peer name is hotcpc12533 -23814> 2013-04-03 10:33:48.320844 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 existing quorum 0,2 -23813> 2013-04-03 10:33:48.320846 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 peer paxos version 427188 vs my version 427188 (ok) -23812> 2013-04-03 10:33:48.320849 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 start_election -23811> 2013-04-03 10:33:48.320850 7fcdffcdb700 10 mon.hotcpc9039@0(probing) e5 cancel_probe_timeout 0x1b91350 -23810> 2013-04-03 10:33:48.320872 7fcdffcdb700 0 log [INF] : mon.hotcpc9039 calling new monitor election -23809> 2013-04-03 10:33:48.320874 7fcdffcdb700 10 send_log to self -23808> 2013-04-03 10:33:48.320876 7fcdffcdb700 10 log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1 -23807> 2013-04-03 10:33:48.320880 7fcdffcdb700 10 will send 2013-04-03 10:33:48.320874 mon.0 10.252.24.25:6789/0 1 : [INF] mon.hotcpc9039 calling new monitor election -23806> 2013-04-03 10:33:48.320892 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x21f8a00 -23805> 2013-04-03 10:33:48.320906 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(0) start -- can i be leader? -23804> 2013-04-03 10:33:48.320980 7fcdffcdb700 1 mon.hotcpc9039@0(electing).elector(350) init, last seen epoch 350 -23803> 2013-04-03 10:33:48.320985 7fcdffcdb700 10 mon.hotcpc9039@0(electing).elector(350) bump_epoch 350 to 351 -23802> 2013-04-03 10:33:48.321347 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 propose 351) v3 -- ?+0 0x20ad480 -23801> 2013-04-03 10:33:48.321365 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 propose 351) v3 -- ?+0 0x20ad900 -23800> 2013-04-03 10:33:48.321381 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x21f8a00 con 0x1bac160 -23799> 2013-04-03 10:33:48.321390 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 do not have session, making new one -23798> 2013-04-03 10:33:48.321394 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 ms_dispatch new session MonSession: mon.0 10.252.24.25:6789/0 is open for mon.0 10.252.24.25:6789/0 -23797> 2013-04-03 10:33:48.321398 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 setting monitor caps on this connection -23796> 2013-04-03 10:33:48.321429 7fcdffcdb700 1 mon.hotcpc9039@0(electing).paxos(paxos recovering c 427161..427188) is_readable now=2013-04-03 10:33:48.321430 lease_expire=0.000000 has v0 lc 427188 -23795> 2013-04-03 10:33:48.373498 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112834 ==== election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 ack 351) v3 ==== 537+0+0 (9292020 0 0) 0x20ad240 con 0x208a840 -23794> 2013-04-03 10:33:48.373542 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(351) handle_ack from mon.2 -23793> 2013-04-03 10:33:48.373548 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(351) so far i have {0=4294967295,2=4294967295} -23792> 2013-04-03 10:33:48.720309 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112835 ==== forward(pg_stats(49 pgs tid 30428 v 596) v1) to leader v1 ==== 18661+0+0 (3860459340 0 0) 0x1bae080 con 0x208a840 -23791> 2013-04-03 10:33:48.720343 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -23790> 2013-04-03 10:33:48.720356 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 mesg 0x20add80 from 10.252.24.88:6789/0 -23789> 2013-04-03 10:33:48.721112 7fcdffcdb700 1 mon.hotcpc9039@0(electing).paxos(paxos recovering c 427161..427188) is_readable now=2013-04-03 10:33:48.721114 lease_expire=0.000000 has v0 lc 427188 -23788> 2013-04-03 10:33:48.780929 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112836 ==== forward(pg_stats(4 pgs tid 401 v 596) v1) to leader v1 ==== 1876+0+0 (1501979510 0 0) 0x1bae600 con 0x208a840 -23787> 2013-04-03 10:33:48.780953 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -23786> 2013-04-03 10:33:48.780964 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -23785> 2013-04-03 10:33:48.781008 7fcdffcdb700 1 mon.hotcpc9039@0(electing).paxos(paxos recovering c 427161..427188) is_readable now=2013-04-03 10:33:48.781009 lease_expire=0.000000 has v0 lc 427188 -23784> 2013-04-03 10:33:49.821578 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112837 ==== forward(pg_stats(5 pgs tid 520 v 596) v1) to leader v1 ==== 2249+0+0 (3344847855 0 0) 0x1bae340 con 0x208a840 -23783> 2013-04-03 10:33:49.821606 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 -23782> 2013-04-03 10:33:49.821617 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 mesg 0x20fc480 from 10.252.24.88:6789/0 -23781> 2013-04-03 10:33:49.821669 7fcdffcdb700 1 mon.hotcpc9039@0(electing).paxos(paxos recovering c 427161..427188) is_readable now=2013-04-03 10:33:49.821669 lease_expire=0.000000 has v0 lc 427188 -23780> 2013-04-03 10:33:52.477128 7fcdfecd9700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x20f4000 sd=22 :6789 s=0 pgs=0 cs=0 l=0).accept sd=22 10.252.58.31:44156/0 -23779> 2013-04-03 10:33:52.477502 7fcdfecd9700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.58.31:6800/10908 mds protocol 0 -23778> 2013-04-03 10:33:52.477837 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 1 ==== auth(proto 0 36 bytes epoch 5) v1 ==== 66+0+0 (2097609634 0 0) 0x20fb000 con 0x2118420 -23777> 2013-04-03 10:33:52.477863 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 36 bytes epoch 5) v1 until we get in quorum -23776> 2013-04-03 10:33:53.318324 7fce004dc700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 36 bytes epoch 5) v1 until we get in quorum -23775> 2013-04-03 10:33:53.321435 7fce004dc700 5 mon.hotcpc9039@0(electing).elector(351) election timer expired -23774> 2013-04-03 10:33:53.321444 7fce004dc700 10 mon.hotcpc9039@0(electing).elector(351) bump_epoch 351 to 352 -23773> 2013-04-03 10:33:53.331805 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 victory 352) v3 -- ?+0 0x20fc6c0 -23772> 2013-04-03 10:33:53.331846 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 win_election, epoch 352 quorum is 0,2 features are 4294967295 -23771> 2013-04-03 10:33:53.331861 7fce004dc700 0 log [INF] : mon.hotcpc9039@0 won leader election with quorum 0,2 -23770> 2013-04-03 10:33:53.331863 7fce004dc700 10 send_log to self -23769> 2013-04-03 10:33:53.331866 7fce004dc700 10 log_queue is 2 last_log 2 sent 1 num 2 unsent 1 sending 1 -23768> 2013-04-03 10:33:53.331870 7fce004dc700 10 will send 2013-04-03 10:33:48.320874 mon.0 10.252.24.25:6789/0 1 : [INF] mon.hotcpc9039 calling new monitor election -23767> 2013-04-03 10:33:53.331879 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x20fb200 -23766> 2013-04-03 10:33:53.331895 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x20fb200 con 0x1bac160 -23765> 2013-04-03 10:33:53.332146 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(collect lc 427188 fc 427161 pn 17600 opn 0) v3 -- ?+0 0x20f4280 -23764> 2013-04-03 10:33:53.332187 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish -23763> 2013-04-03 10:33:53.332193 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -23762> 2013-04-03 10:33:53.332196 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: mds.? 10.252.58.31:6800/10908 is open for mds.? 10.252.58.31:6800/10908 -23761> 2013-04-03 10:33:53.332201 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -23760> 2013-04-03 10:33:53.332235 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427161..427188) is_readable now=2013-04-03 10:33:53.332236 lease_expire=0.000000 has v0 lc 427188 -23759> 2013-04-03 10:33:53.332245 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 resend_routed_requests -23758> 2013-04-03 10:33:53.332249 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 register_cluster_logger -23757> 2013-04-03 10:33:53.332255 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start -23756> 2013-04-03 10:33:53.332256 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round curr 0 -23755> 2013-04-03 10:33:53.332258 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round new 1 -23754> 2013-04-03 10:33:53.332260 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck -23753> 2013-04-03 10:33:53.332262 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck start timecheck epoch 352 round 1 -23752> 2013-04-03 10:33:53.332266 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck send time_check( ping e 352 r 1 ) v1 to mon.2 10.252.24.88:6789/0 -23751> 2013-04-03 10:33:53.332271 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- time_check( ping e 352 r 1 ) v1 -- ?+0 0x20fcb40 -23750> 2013-04-03 10:33:53.332277 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round setting up next event -23749> 2013-04-03 10:33:53.332341 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427161..427188) is_readable now=2013-04-03 10:33:53.332344 lease_expire=0.000000 has v0 lc 427188 -23748> 2013-04-03 10:33:53.360966 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112838 ==== forward(mdsbeacon(11234/hotcpc9039 up:boot seq 1 v0) v2) to leader v1 ==== 560+0+0 (2255487676 0 0) 0x1bae8c0 con 0x208a840 -23747> 2013-04-03 10:33:53.360982 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.? 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -23746> 2013-04-03 10:33:53.360991 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae600 from 10.252.24.88:6789/0 -23745> 2013-04-03 10:33:53.361021 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427161..427188) is_readable now=2013-04-03 10:33:53.361022 lease_expire=0.000000 has v0 lc 427188 -23744> 2013-04-03 10:33:53.361033 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112839 ==== forward(pg_stats(1 pgs tid 519 v 596) v1) to leader v1 ==== 757+0+0 (568775297 0 0) 0x1bae080 con 0x208a840 -23743> 2013-04-03 10:33:53.361041 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 -23742> 2013-04-03 10:33:53.361047 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc6c0 from 10.252.24.88:6789/0 -23741> 2013-04-03 10:33:53.361071 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427161..427188) is_readable now=2013-04-03 10:33:53.361071 lease_expire=0.000000 has v0 lc 427188 -23740> 2013-04-03 10:33:53.361093 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112840 ==== forward(osd_boot(osd.1 booted 0 v596) v3) to leader v1 ==== 898+0+0 (406753043 0 0) 0x1baeb80 con 0x208a840 -23739> 2013-04-03 10:33:53.361102 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -23738> 2013-04-03 10:33:53.361108 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20c5400 from 10.252.24.88:6789/0 -23737> 2013-04-03 10:33:53.361150 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427161..427188) is_readable now=2013-04-03 10:33:53.361151 lease_expire=0.000000 has v0 lc 427188 -23736> 2013-04-03 10:33:53.362978 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112841 ==== forward(pg_stats(49 pgs tid 30427 v 596) v1) to leader v1 ==== 18661+0+0 (1222095414 0 0) 0x1bae080 con 0x208a840 -23735> 2013-04-03 10:33:53.362991 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -23734> 2013-04-03 10:33:53.362998 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 -23733> 2013-04-03 10:33:53.363025 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427161..427188) is_readable now=2013-04-03 10:33:53.363026 lease_expire=0.000000 has v0 lc 427188 -23732> 2013-04-03 10:33:53.363037 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112842 ==== forward(mdsbeacon(11234/hotcpc9039 up:replay seq 2 v184) v2) to leader v1 ==== 560+0+0 (2098651222 0 0) 0x1bae8c0 con 0x208a840 -23731> 2013-04-03 10:33:53.363045 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -23730> 2013-04-03 10:33:53.363053 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae340 from 10.252.24.88:6789/0 -23729> 2013-04-03 10:33:53.363078 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427161..427188) is_readable now=2013-04-03 10:33:53.363078 lease_expire=0.000000 has v0 lc 427188 -23728> 2013-04-03 10:33:53.364584 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112843 ==== forward(pg_stats(49 pgs tid 30428 v 596) v1) to leader v1 ==== 18661+0+0 (3860459340 0 0) 0x25f0dc0 con 0x208a840 -23727> 2013-04-03 10:33:53.364593 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -23726> 2013-04-03 10:33:53.364599 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fcd80 from 10.252.24.88:6789/0 -23725> 2013-04-03 10:33:53.364625 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427161..427188) is_readable now=2013-04-03 10:33:53.364626 lease_expire=0.000000 has v0 lc 427188 -23724> 2013-04-03 10:33:53.364730 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112844 ==== forward(pg_stats(4 pgs tid 401 v 596) v1) to leader v1 ==== 1876+0+0 (1501979510 0 0) 0x25f0b00 con 0x208a840 -23723> 2013-04-03 10:33:53.364739 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -23722> 2013-04-03 10:33:53.364746 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1c18900 from 10.252.24.88:6789/0 -23721> 2013-04-03 10:33:53.364771 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427161..427188) is_readable now=2013-04-03 10:33:53.364772 lease_expire=0.000000 has v0 lc 427188 -23720> 2013-04-03 10:33:53.364819 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112845 ==== forward(pg_stats(5 pgs tid 520 v 596) v1) to leader v1 ==== 2249+0+0 (3344847855 0 0) 0x25f0840 con 0x208a840 -23719> 2013-04-03 10:33:53.364828 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 -23718> 2013-04-03 10:33:53.364836 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1c186c0 from 10.252.24.88:6789/0 -23717> 2013-04-03 10:33:53.364858 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427161..427188) is_readable now=2013-04-03 10:33:53.364859 lease_expire=0.000000 has v0 lc 427188 -23716> 2013-04-03 10:33:53.372640 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112846 ==== paxos(last lc 427188 fc 427161 pn 17600 opn 0) v3 ==== 80+0+0 (3976057819 0 0) 0x20f4280 con 0x208a840 -23715> 2013-04-03 10:33:53.372669 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427188 fc 427161 pn 0 opn 0) v3 -- ?+0 0x1f91c80 -23714> 2013-04-03 10:33:53.372856 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 create_pending v 28958 -23713> 2013-04-03 10:33:53.372874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 check_osd_map already seen 596 >= 596 -23712> 2013-04-03 10:33:53.372878 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 update_logger -23711> 2013-04-03 10:33:53.372895 7fcdffcdb700 0 log [INF] : pgmap v28957: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 4872B/s wr, 0op/s -23710> 2013-04-03 10:33:53.372898 7fcdffcdb700 10 send_log to self -23709> 2013-04-03 10:33:53.372899 7fcdffcdb700 10 log_queue is 3 last_log 3 sent 1 num 3 unsent 2 sending 2 -23708> 2013-04-03 10:33:53.372900 7fcdffcdb700 10 will send 2013-04-03 10:33:48.320874 mon.0 10.252.24.25:6789/0 1 : [INF] mon.hotcpc9039 calling new monitor election -23707> 2013-04-03 10:33:53.372908 7fcdffcdb700 10 will send 2013-04-03 10:33:53.331862 mon.0 10.252.24.25:6789/0 2 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 -23706> 2013-04-03 10:33:53.372911 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x25ef000 -23705> 2013-04-03 10:33:53.372942 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 create_pending e185 -23704> 2013-04-03 10:33:53.372961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby -23703> 2013-04-03 10:33:53.372972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 adding 10.252.58.31:6800/10908 mds.-1.0 up:standby to last_beacon -23702> 2013-04-03 10:33:53.372977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 adding 10.252.24.88:6800/17213 mds.-1.0 up:standby to last_beacon -23701> 2013-04-03 10:33:53.372980 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 adding 10.252.24.25:6802/3144 mds.0.29 up:replay to last_beacon -23700> 2013-04-03 10:33:53.372986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 update_logger -23699> 2013-04-03 10:33:53.372992 7fcdffcdb700 0 log [INF] : mdsmap e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby -23698> 2013-04-03 10:33:53.372993 7fcdffcdb700 10 send_log to self -23697> 2013-04-03 10:33:53.372994 7fcdffcdb700 10 log_queue is 4 last_log 4 sent 2 num 4 unsent 2 sending 2 -23696> 2013-04-03 10:33:53.372995 7fcdffcdb700 10 will send 2013-04-03 10:33:53.331862 mon.0 10.252.24.25:6789/0 2 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 -23695> 2013-04-03 10:33:53.372999 7fcdffcdb700 10 will send 2013-04-03 10:33:53.372898 mon.0 10.252.24.25:6789/0 3 : [INF] pgmap v28957: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 4872B/s wr, 0op/s -23694> 2013-04-03 10:33:53.373006 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2513a00 -23693> 2013-04-03 10:33:53.373056 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e596 create_pending e 597 -23692> 2013-04-03 10:33:53.373063 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e596 remove_redundant_pg_temp -23691> 2013-04-03 10:33:53.373065 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e596 remove_down_pg_temp -23690> 2013-04-03 10:33:53.373098 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e596 update_logger -23689> 2013-04-03 10:33:53.373107 7fcdffcdb700 0 log [INF] : osdmap e596: 5 osds: 5 up, 5 in -23688> 2013-04-03 10:33:53.373108 7fcdffcdb700 10 send_log to self -23687> 2013-04-03 10:33:53.373109 7fcdffcdb700 10 log_queue is 5 last_log 5 sent 3 num 5 unsent 2 sending 2 -23686> 2013-04-03 10:33:53.373110 7fcdffcdb700 10 will send 2013-04-03 10:33:53.372898 mon.0 10.252.24.25:6789/0 3 : [INF] pgmap v28957: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 4872B/s wr, 0op/s -23685> 2013-04-03 10:33:53.373116 7fcdffcdb700 10 will send 2013-04-03 10:33:53.372993 mon.0 10.252.24.25:6789/0 4 : [INF] mdsmap e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby -23684> 2013-04-03 10:33:53.373121 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2513800 -23683> 2013-04-03 10:33:53.373129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos -23682> 2013-04-03 10:33:53.373150 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 -23681> 2013-04-03 10:33:53.373181 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 create_pending v 393966 -23680> 2013-04-03 10:33:53.373219 7fcdffcdb700 10 mon.hotcpc9039@0(leader).monmap v5 create_pending monmap epoch 6 -23679> 2013-04-03 10:33:53.373242 7fcdffcdb700 0 log [INF] : monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -23678> 2013-04-03 10:33:53.373245 7fcdffcdb700 10 send_log to self -23677> 2013-04-03 10:33:53.373246 7fcdffcdb700 10 log_queue is 6 last_log 6 sent 4 num 6 unsent 2 sending 2 -23676> 2013-04-03 10:33:53.373247 7fcdffcdb700 10 will send 2013-04-03 10:33:53.372993 mon.0 10.252.24.25:6789/0 4 : [INF] mdsmap e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby -23675> 2013-04-03 10:33:53.373251 7fcdffcdb700 10 will send 2013-04-03 10:33:53.373108 mon.0 10.252.24.25:6789/0 5 : [INF] osdmap e596: 5 osds: 5 up, 5 in -23674> 2013-04-03 10:33:53.373255 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2513600 -23673> 2013-04-03 10:33:53.373264 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -23672> 2013-04-03 10:33:53.373292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 create_pending v 2060 -23671> 2013-04-03 10:33:53.373323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 AuthMonitor::on_active() -23670> 2013-04-03 10:33:53.373351 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos -23669> 2013-04-03 10:33:53.373365 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 -23668> 2013-04-03 10:33:53.373375 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -23667> 2013-04-03 10:33:53.373388 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_log log(1 entries) v1 from mon.0 -23666> 2013-04-03 10:33:53.373407 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -23665> 2013-04-03 10:33:53.373419 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_log log(1 entries) v1 from mon.0 -23664> 2013-04-03 10:33:53.373428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:48.320874 mon.0 10.252.24.25:6789/0 1 : [INF] mon.hotcpc9039 calling new monitor election -23663> 2013-04-03 10:33:53.373475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(49 pgs tid 30428 v 596) v1 from osd.3 10.252.24.88:6801/17538 -23662> 2013-04-03 10:33:53.373484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(49 pgs tid 30428 v 596) v1 from osd.3 10.252.24.88:6801/17538 -23661> 2013-04-03 10:33:53.373489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(49 pgs tid 30428 v 596) v1 from osd.3 -23660> 2013-04-03 10:33:53.373499 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.3 osd_stat(1187 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1186 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -23659> 2013-04-03 10:33:53.373601 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(4 pgs tid 401 v 596) v1 from osd.0 10.252.58.31:6802/2682 -23658> 2013-04-03 10:33:53.373611 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(4 pgs tid 401 v 596) v1 from osd.0 10.252.58.31:6802/2682 -23657> 2013-04-03 10:33:53.373614 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(4 pgs tid 401 v 596) v1 from osd.0 -23656> 2013-04-03 10:33:53.373618 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.0 osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -23655> 2013-04-03 10:33:53.373655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(5 pgs tid 520 v 596) v1 from osd.2 10.252.24.30:6801/10373 -23654> 2013-04-03 10:33:53.373666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(5 pgs tid 520 v 596) v1 from osd.2 10.252.24.30:6801/10373 -23653> 2013-04-03 10:33:53.373669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(5 pgs tid 520 v 596) v1 from osd.2 -23652> 2013-04-03 10:33:53.373673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -23651> 2013-04-03 10:33:53.373706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -23650> 2013-04-03 10:33:53.373724 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 preprocess_query auth(proto 0 36 bytes epoch 5) v1 from mds.? 10.252.58.31:6800/10908 -23649> 2013-04-03 10:33:53.373739 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 prep_auth() blob_size=36 -23648> 2013-04-03 10:33:53.373775 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x211f400 con 0x2118420 -23647> 2013-04-03 10:33:53.373816 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos -23646> 2013-04-03 10:33:53.373831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 -23645> 2013-04-03 10:33:53.373839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -23644> 2013-04-03 10:33:53.373851 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_log log(1 entries) v1 from mon.0 -23643> 2013-04-03 10:33:53.373866 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 nothing new -23642> 2013-04-03 10:33:53.373895 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:boot seq 1 v0) v2 from mds.? 10.252.24.25:6802/3144 -23641> 2013-04-03 10:33:53.373905 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 mds_beacon mdsbeacon(11234/hotcpc9039 up:boot seq 1 v0) v2 ignoring requested state, because mds hasn't seen latest map -23640> 2013-04-03 10:33:53.373909 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:boot seq 1 v184) v2 tid 17355) v2 -- ?+0 0x20ad240 con 0x208a840 -23639> 2013-04-03 10:33:53.373940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(1 pgs tid 519 v 596) v1 from osd.2 10.252.24.30:6801/10373 -23638> 2013-04-03 10:33:53.373947 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(1 pgs tid 519 v 596) v1 from osd.2 10.252.24.30:6801/10373 -23637> 2013-04-03 10:33:53.373951 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(1 pgs tid 519 v 596) v1 from osd.2 -23636> 2013-04-03 10:33:53.373954 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -23635> 2013-04-03 10:33:53.373993 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e596 preprocess_query osd_boot(osd.1 booted 0 v596) v3 from osd.1 10.252.24.25:6805/3572 -23634> 2013-04-03 10:33:53.374004 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e596 preprocess_boot from osd.1 10.252.24.25:6805/3572 -23633> 2013-04-03 10:33:53.374008 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e596 prepare_update osd_boot(osd.1 booted 0 v596) v3 from osd.1 10.252.24.25:6805/3572 -23632> 2013-04-03 10:33:53.374012 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e596 prepare_boot from osd.1 10.252.24.25:6805/3572 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.1 25c71ceb-f510-49a4-bf1a-ba1f96204778 e596 [95,596] lci=[593,596]) cluster_addr 10.252.24.25:6806/3572 hb_addr 10.252.24.25:6807/3572 -23631> 2013-04-03 10:33:53.374022 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e596 prepare_boot was up, first marking down osd.1 10.252.24.25:6801/4077 -23630> 2013-04-03 10:33:53.374028 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e596 should_propose -23629> 2013-04-03 10:33:53.374054 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(49 pgs tid 30427 v 596) v1 from osd.3 10.252.24.88:6801/17538 -23628> 2013-04-03 10:33:53.374060 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(49 pgs tid 30427 v 596) v1 from osd.3 10.252.24.88:6801/17538 -23627> 2013-04-03 10:33:53.374064 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(49 pgs tid 30427 v 596) v1 from osd.3 -23626> 2013-04-03 10:33:53.374066 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.3 osd_stat(1187 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1186 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -23625> 2013-04-03 10:33:53.374131 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 2 v184) v2 from mds.0 10.252.24.25:6802/3144 -23624> 2013-04-03 10:33:53.374141 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:replay seq 2 v184) v2 tid 17360) v2 -- ?+0 0x20fcb40 con 0x208a840 -23623> 2013-04-03 10:33:53.374169 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(49 pgs tid 30428 v 596) v1 from osd.3 10.252.24.88:6801/17538 -23622> 2013-04-03 10:33:53.374175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(49 pgs tid 30428 v 596) v1 from osd.3 10.252.24.88:6801/17538 -23621> 2013-04-03 10:33:53.374179 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(49 pgs tid 30428 v 596) v1 from osd.3 -23620> 2013-04-03 10:33:53.374182 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.3 osd_stat(1187 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1186 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -23619> 2013-04-03 10:33:53.374241 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(4 pgs tid 401 v 596) v1 from osd.0 10.252.58.31:6802/2682 -23618> 2013-04-03 10:33:53.374249 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(4 pgs tid 401 v 596) v1 from osd.0 10.252.58.31:6802/2682 -23617> 2013-04-03 10:33:53.374252 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(4 pgs tid 401 v 596) v1 from osd.0 -23616> 2013-04-03 10:33:53.374256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.0 osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -23615> 2013-04-03 10:33:53.374286 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 preprocess_query pg_stats(5 pgs tid 520 v 596) v1 from osd.2 10.252.24.30:6801/10373 -23614> 2013-04-03 10:33:53.374293 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_update pg_stats(5 pgs tid 520 v 596) v1 from osd.2 10.252.24.30:6801/10373 -23613> 2013-04-03 10:33:53.374296 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 prepare_pg_stats pg_stats(5 pgs tid 520 v 596) v1 from osd.2 -23612> 2013-04-03 10:33:53.374299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28957 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -23611> 2013-04-03 10:33:53.374331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos -23610> 2013-04-03 10:33:53.374346 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 -23609> 2013-04-03 10:33:53.374362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -23608> 2013-04-03 10:33:53.374381 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112847 ==== time_check( pong e 352 r 1 ts 2013-04-03 10:33:53.371770 ) v1 ==== 36+0+0 (838653692 0 0) 0x1c18480 con 0x208a840 -23607> 2013-04-03 10:33:53.374394 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck time_check( pong e 352 r 1 ts 2013-04-03 10:33:53.371770 ) v1 -23606> 2013-04-03 10:33:53.374401 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader time_check( pong e 352 r 1 ts 2013-04-03 10:33:53.371770 ) v1 -23605> 2013-04-03 10:33:53.374408 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader from mon.2 10.252.24.88:6789/0 ts 2013-04-03 10:33:53.371770 delta -0.002635 skew_bound 0 latency 0.04214 -23604> 2013-04-03 10:33:53.374422 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader got pongs from everybody (2 total) -23603> 2013-04-03 10:33:53.374425 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish_round curr 1 -23602> 2013-04-03 10:33:53.374427 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_report -23601> 2013-04-03 10:33:53.374430 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_report send report to mon.2 10.252.24.88:6789/0 -23600> 2013-04-03 10:33:53.374433 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- time_check( report e 352 r 2 #skews 2 #latencies 2 ) v1 -- ?+0 0x20fc000 -23599> 2013-04-03 10:33:53.374441 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x25ef000 con 0x1bac160 -23598> 2013-04-03 10:33:53.374464 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos -23597> 2013-04-03 10:33:53.374483 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 -23596> 2013-04-03 10:33:53.374492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23595> 2013-04-03 10:33:53.374503 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_log log(2 entries) v1 from mon.0 -23594> 2013-04-03 10:33:53.374526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23593> 2013-04-03 10:33:53.374537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_log log(2 entries) v1 from mon.0 -23592> 2013-04-03 10:33:53.374546 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:48.320874 mon.0 10.252.24.25:6789/0 1 : [INF] mon.hotcpc9039 calling new monitor election -23591> 2013-04-03 10:33:53.374565 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:53.331862 mon.0 10.252.24.25:6789/0 2 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 -23590> 2013-04-03 10:33:53.374593 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112848 ==== paxos(lease_ack lc 427188 fc 427161 pn 0 opn 0) v3 ==== 80+0+0 (2130747282 0 0) 0x1f91c80 con 0x208a840 -23589> 2013-04-03 10:33:53.374626 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos -23588> 2013-04-03 10:33:53.374640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 -23587> 2013-04-03 10:33:53.374655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -23586> 2013-04-03 10:33:53.374672 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2513a00 con 0x1bac160 -23585> 2013-04-03 10:33:53.374692 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos -23584> 2013-04-03 10:33:53.374705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 -23583> 2013-04-03 10:33:53.374712 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23582> 2013-04-03 10:33:53.374722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_log log(2 entries) v1 from mon.0 -23581> 2013-04-03 10:33:53.374745 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23580> 2013-04-03 10:33:53.374756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_log log(2 entries) v1 from mon.0 -23579> 2013-04-03 10:33:53.374764 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:53.331862 mon.0 10.252.24.25:6789/0 2 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 -23578> 2013-04-03 10:33:53.374784 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:53.372898 mon.0 10.252.24.25:6789/0 3 : [INF] pgmap v28957: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 4872B/s wr, 0op/s -23577> 2013-04-03 10:33:53.374813 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2513800 con 0x1bac160 -23576> 2013-04-03 10:33:53.374836 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos -23575> 2013-04-03 10:33:53.374850 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 -23574> 2013-04-03 10:33:53.374858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23573> 2013-04-03 10:33:53.374869 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_log log(2 entries) v1 from mon.0 -23572> 2013-04-03 10:33:53.374891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23571> 2013-04-03 10:33:53.374902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_log log(2 entries) v1 from mon.0 -23570> 2013-04-03 10:33:53.374910 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:53.372898 mon.0 10.252.24.25:6789/0 3 : [INF] pgmap v28957: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 4872B/s wr, 0op/s -23569> 2013-04-03 10:33:53.374929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:53.372993 mon.0 10.252.24.25:6789/0 4 : [INF] mdsmap e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby -23568> 2013-04-03 10:33:53.374958 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2513600 con 0x1bac160 -23567> 2013-04-03 10:33:53.374977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos -23566> 2013-04-03 10:33:53.374990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 update_from_paxos version 393965 summary v 393965 -23565> 2013-04-03 10:33:53.374998 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23564> 2013-04-03 10:33:53.375009 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 preprocess_log log(2 entries) v1 from mon.0 -23563> 2013-04-03 10:33:53.375032 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23562> 2013-04-03 10:33:53.375042 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 prepare_log log(2 entries) v1 from mon.0 -23561> 2013-04-03 10:33:53.375051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:53.372993 mon.0 10.252.24.25:6789/0 4 : [INF] mdsmap e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby -23560> 2013-04-03 10:33:53.375069 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393965 logging 2013-04-03 10:33:53.373108 mon.0 10.252.24.25:6789/0 5 : [INF] osdmap e596: 5 osds: 5 up, 5 in -23559> 2013-04-03 10:33:53.375103 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (771627134 0 0) 0x209c200 con 0x2118420 -23558> 2013-04-03 10:33:53.375124 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -23557> 2013-04-03 10:33:53.375139 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 preprocess_query auth(proto 2 144 bytes epoch 0) v1 from mds.? 10.252.58.31:6800/10908 -23556> 2013-04-03 10:33:53.375150 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 prep_auth() blob_size=144 -23555> 2013-04-03 10:33:53.375297 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x25ef800 con 0x2118420 -23554> 2013-04-03 10:33:53.375877 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 3 ==== mon_subscribe({mdsmap=185+,monmap=6+}) v2 ==== 42+0+0 (945815558 0 0) 0x2224000 con 0x2118420 -23553> 2013-04-03 10:33:53.375912 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=185+,monmap=6+}) v2 -23552> 2013-04-03 10:33:53.375921 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -23551> 2013-04-03 10:33:53.375923 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.58.31:6800/10908 -- mon_subscribe_ack(300s) v1 -- ?+0 0x22948c0 -23550> 2013-04-03 10:33:53.381962 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112849 ==== forward(mdsbeacon(11234/hotcpc9039 up:replay seq 3 v184) v2) to leader v1 ==== 560+0+0 (234235230 0 0) 0x1bae600 con 0x208a840 -23549> 2013-04-03 10:33:53.381973 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -23548> 2013-04-03 10:33:53.381980 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x25f0840 from 10.252.24.88:6789/0 -23547> 2013-04-03 10:33:53.382008 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 3 v184) v2 from mds.0 10.252.24.25:6802/3144 -23546> 2013-04-03 10:33:53.382017 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:replay seq 3 v184) v2 tid 17364) v2 -- ?+0 0x1c18480 con 0x208a840 -23545> 2013-04-03 10:33:53.423616 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393965 encode_pending v393966 -23544> 2013-04-03 10:33:53.423645 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427161..427188) queue_proposal bl 1321 bytes; ctx = 0x1b91350 -23543> 2013-04-03 10:33:53.423653 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427161..427188) propose_queued 427189 1321 bytes -23542> 2013-04-03 10:33:53.423932 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427188 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4280 -23541> 2013-04-03 10:33:53.424033 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28957 encode_pending v 28958 -23540> 2013-04-03 10:33:53.424204 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427161..427188) queue_proposal bl 22047 bytes; ctx = 0x1b91590 -23539> 2013-04-03 10:33:53.424212 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427161..427188) propose_new_value not active; proposal queued -23538> 2013-04-03 10:33:53.424265 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e596 encode_pending e 597 -23537> 2013-04-03 10:33:53.424273 7fce004dc700 2 mon.hotcpc9039@0(leader).osd e596 osd.1 DOWN -23536> 2013-04-03 10:33:53.424305 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427161..427188) queue_proposal bl 214 bytes; ctx = 0x1b91540 -23535> 2013-04-03 10:33:53.424311 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427161..427188) propose_new_value not active; proposal queued -23534> 2013-04-03 10:33:53.456174 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112850 ==== paxos(accept lc 427188 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2045281353 0 0) 0x20f4280 con 0x208a840 -23533> 2013-04-03 10:33:53.458367 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427189 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4780 -23532> 2013-04-03 10:33:53.458405 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427189 fc 427161 pn 0 opn 0) v3 -- ?+0 0x20f4500 -23531> 2013-04-03 10:33:53.458440 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos -23530> 2013-04-03 10:33:53.458464 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos version 393966 summary v 393965 -23529> 2013-04-03 10:33:53.458478 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos latest full 393798 -23528> 2013-04-03 10:33:53.458496 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 update_from_paxos applying incremental log 393966 2013-04-03 10:33:48.320874 mon.0 10.252.24.25:6789/0 1 : [INF] mon.hotcpc9039 calling new monitor election -23527> 2013-04-03 10:33:53.458521 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 update_from_paxos applying incremental log 393966 2013-04-03 10:33:53.331862 mon.0 10.252.24.25:6789/0 2 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 -23526> 2013-04-03 10:33:53.458541 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 update_from_paxos applying incremental log 393966 2013-04-03 10:33:53.372898 mon.0 10.252.24.25:6789/0 3 : [INF] pgmap v28957: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 4872B/s wr, 0op/s -23525> 2013-04-03 10:33:53.458560 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 update_from_paxos applying incremental log 393966 2013-04-03 10:33:53.372993 mon.0 10.252.24.25:6789/0 4 : [INF] mdsmap e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby -23524> 2013-04-03 10:33:53.458578 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 update_from_paxos applying incremental log 393966 2013-04-03 10:33:53.373108 mon.0 10.252.24.25:6789/0 5 : [INF] osdmap e596: 5 osds: 5 up, 5 in -23523> 2013-04-03 10:33:53.458611 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 check_subs -23522> 2013-04-03 10:33:53.458635 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 create_pending v 393967 -23521> 2013-04-03 10:33:53.458655 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 _updated_log for mon.0 10.252.24.25:6789/0 -23520> 2013-04-03 10:33:53.458667 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 1) v1 -- ?+0 0x2224000 con 0x1bac160 -23519> 2013-04-03 10:33:53.458675 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 _updated_log for mon.0 10.252.24.25:6789/0 -23518> 2013-04-03 10:33:53.458688 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 2) v1 -- ?+0 0x22248c0 con 0x1bac160 -23517> 2013-04-03 10:33:53.458693 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 _updated_log for mon.0 10.252.24.25:6789/0 -23516> 2013-04-03 10:33:53.458704 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 3) v1 -- ?+0 0x2224700 con 0x1bac160 -23515> 2013-04-03 10:33:53.458709 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 _updated_log for mon.0 10.252.24.25:6789/0 -23514> 2013-04-03 10:33:53.458720 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 4) v1 -- ?+0 0x2224540 con 0x1bac160 -23513> 2013-04-03 10:33:53.458724 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393966 _updated_log for mon.0 10.252.24.25:6789/0 -23512> 2013-04-03 10:33:53.458736 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 5) v1 -- ?+0 0x2224380 con 0x1bac160 -23511> 2013-04-03 10:33:53.458754 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427161..427189) propose_queued 427190 22047 bytes -23510> 2013-04-03 10:33:53.460062 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427189 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4c80 -23509> 2013-04-03 10:33:53.460100 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 1) v1 ==== 0+0+0 (0 0 0) 0x2224000 con 0x1bac160 -23508> 2013-04-03 10:33:53.460107 7fcdffcdb700 10 handle_log_ack log(last 1) v1 -23507> 2013-04-03 10:33:53.460109 7fcdffcdb700 10 logged 2013-04-03 10:33:48.320874 mon.0 10.252.24.25:6789/0 1 : [INF] mon.hotcpc9039 calling new monitor election -23506> 2013-04-03 10:33:53.460116 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 2) v1 ==== 0+0+0 (0 0 0) 0x22248c0 con 0x1bac160 -23505> 2013-04-03 10:33:53.460123 7fcdffcdb700 10 handle_log_ack log(last 2) v1 -23504> 2013-04-03 10:33:53.460124 7fcdffcdb700 10 logged 2013-04-03 10:33:53.331862 mon.0 10.252.24.25:6789/0 2 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 -23503> 2013-04-03 10:33:53.460131 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 3) v1 ==== 0+0+0 (0 0 0) 0x2224700 con 0x1bac160 -23502> 2013-04-03 10:33:53.460136 7fcdffcdb700 10 handle_log_ack log(last 3) v1 -23501> 2013-04-03 10:33:53.460139 7fcdffcdb700 10 logged 2013-04-03 10:33:53.372898 mon.0 10.252.24.25:6789/0 3 : [INF] pgmap v28957: 240 pgs: 240 active+clean; 258 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 4872B/s wr, 0op/s -23500> 2013-04-03 10:33:53.460145 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 4) v1 ==== 0+0+0 (0 0 0) 0x2224540 con 0x1bac160 -23499> 2013-04-03 10:33:53.460151 7fcdffcdb700 10 handle_log_ack log(last 4) v1 -23498> 2013-04-03 10:33:53.460153 7fcdffcdb700 10 logged 2013-04-03 10:33:53.372993 mon.0 10.252.24.25:6789/0 4 : [INF] mdsmap e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby -23497> 2013-04-03 10:33:53.460158 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 5) v1 ==== 0+0+0 (0 0 0) 0x2224380 con 0x1bac160 -23496> 2013-04-03 10:33:53.460164 7fcdffcdb700 10 handle_log_ack log(last 5) v1 -23495> 2013-04-03 10:33:53.460165 7fcdffcdb700 10 logged 2013-04-03 10:33:53.373108 mon.0 10.252.24.25:6789/0 5 : [INF] osdmap e596: 5 osds: 5 up, 5 in -23494> 2013-04-03 10:33:53.468170 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112851 ==== paxos(lease_ack lc 427189 fc 427161 pn 0 opn 0) v3 ==== 80+0+0 (2177995031 0 0) 0x20f4c80 con 0x208a840 -23493> 2013-04-03 10:33:53.493261 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112852 ==== paxos(accept lc 427189 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2730375841 0 0) 0x20f4500 con 0x208a840 -23492> 2013-04-03 10:33:53.494294 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427190 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4c80 -23491> 2013-04-03 10:33:53.494320 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427190 fc 427161 pn 0 opn 0) v3 -- ?+0 0x20f4280 -23490> 2013-04-03 10:33:53.494454 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28957 update_from_paxos applying incremental 28958 -23489> 2013-04-03 10:33:53.494627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 v28958: 240 pgs: 240 active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 1302B/s wr, 13op/s -23488> 2013-04-03 10:33:53.494661 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 send_pg_creates to 0 pgs -23487> 2013-04-03 10:33:53.494666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 update_logger -23486> 2013-04-03 10:33:53.494684 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 create_pending v 28959 -23485> 2013-04-03 10:33:53.494696 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.3 10.252.24.88:6801/17538 -23484> 2013-04-03 10:33:53.494703 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(49 pgs tid 30428) v1 tid 17361) v2 -- ?+0 0x20fc000 con 0x208a840 -23483> 2013-04-03 10:33:53.494720 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.0 10.252.58.31:6802/2682 -23482> 2013-04-03 10:33:53.494724 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(4 pgs tid 401) v1 tid 17362) v2 -- ?+0 0x20add80 con 0x208a840 -23481> 2013-04-03 10:33:53.494757 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.2 10.252.24.30:6801/10373 -23480> 2013-04-03 10:33:53.494762 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(5 pgs tid 520) v1 tid 17363) v2 -- ?+0 0x20adb40 con 0x208a840 -23479> 2013-04-03 10:33:53.494784 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.2 10.252.24.30:6801/10373 -23478> 2013-04-03 10:33:53.494790 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 519) v1 tid 17357) v2 -- ?+0 0x20fc480 con 0x208a840 -23477> 2013-04-03 10:33:53.494811 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.3 10.252.24.88:6801/17538 -23476> 2013-04-03 10:33:53.494816 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(49 pgs tid 30427) v1 tid 17359) v2 -- ?+0 0x20fc6c0 con 0x208a840 -23475> 2013-04-03 10:33:53.494854 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.3 10.252.24.88:6801/17538 -23474> 2013-04-03 10:33:53.494860 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(49 pgs tid 30428) v1 tid 17361) v2 -- ?+0 0x20fc240 con 0x208a840 -23473> 2013-04-03 10:33:53.494887 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.0 10.252.58.31:6802/2682 -23472> 2013-04-03 10:33:53.494893 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(4 pgs tid 401) v1 tid 17362) v2 -- ?+0 0x20fcd80 con 0x208a840 -23471> 2013-04-03 10:33:53.494918 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 _updated_stats for osd.2 10.252.24.30:6801/10373 -23470> 2013-04-03 10:33:53.494923 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(5 pgs tid 520) v1 tid 17363) v2 -- ?+0 0x1c18900 con 0x208a840 -23469> 2013-04-03 10:33:53.494945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 check_osd_map already seen 596 >= 596 -23468> 2013-04-03 10:33:53.494949 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 update_logger -23467> 2013-04-03 10:33:53.494959 7fcdffcdb700 0 log [INF] : pgmap v28958: 240 pgs: 240 active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 1302B/s wr, 13op/s -23466> 2013-04-03 10:33:53.494960 7fcdffcdb700 10 send_log to self -23465> 2013-04-03 10:33:53.494961 7fcdffcdb700 10 log_queue is 2 last_log 7 sent 5 num 2 unsent 2 sending 2 -23464> 2013-04-03 10:33:53.494962 7fcdffcdb700 10 will send 2013-04-03 10:33:53.373245 mon.0 10.252.24.25:6789/0 6 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -23463> 2013-04-03 10:33:53.494968 7fcdffcdb700 10 will send 2013-04-03 10:33:53.494960 mon.0 10.252.24.25:6789/0 7 : [INF] pgmap v28958: 240 pgs: 240 active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 1302B/s wr, 13op/s -23462> 2013-04-03 10:33:53.494974 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x25ef200 -23461> 2013-04-03 10:33:53.494999 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427161..427190) propose_queued 427191 214 bytes -23460> 2013-04-03 10:33:53.495606 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427190 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4a00 -23459> 2013-04-03 10:33:53.495649 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x25ef200 con 0x1bac160 -23458> 2013-04-03 10:33:53.495679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos -23457> 2013-04-03 10:33:53.495694 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos version 393966 summary v 393966 -23456> 2013-04-03 10:33:53.495702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23455> 2013-04-03 10:33:53.495713 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 preprocess_log log(2 entries) v1 from mon.0 -23454> 2013-04-03 10:33:53.495738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23453> 2013-04-03 10:33:53.495749 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 prepare_log log(2 entries) v1 from mon.0 -23452> 2013-04-03 10:33:53.495759 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 logging 2013-04-03 10:33:53.373245 mon.0 10.252.24.25:6789/0 6 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -23451> 2013-04-03 10:33:53.495779 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 logging 2013-04-03 10:33:53.494960 mon.0 10.252.24.25:6789/0 7 : [INF] pgmap v28958: 240 pgs: 240 active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 1302B/s wr, 13op/s -23450> 2013-04-03 10:33:53.526858 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112853 ==== paxos(lease_ack lc 427190 fc 427161 pn 0 opn 0) v3 ==== 80+0+0 (1127233749 0 0) 0x20f4a00 con 0x208a840 -23449> 2013-04-03 10:33:53.539921 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112854 ==== paxos(accept lc 427190 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1218007562 0 0) 0x20f4280 con 0x208a840 -23448> 2013-04-03 10:33:53.540177 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427191 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4a00 -23447> 2013-04-03 10:33:53.540209 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427191 fc 427161 pn 0 opn 0) v3 -- ?+0 0x20f4500 -23446> 2013-04-03 10:33:53.540285 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e596 update_from_paxos applying incremental 597 -23445> 2013-04-03 10:33:53.540337 7fcdffcdb700 1 mon.hotcpc9039@0(leader).osd e597 e597: 5 osds: 4 up, 5 in -23444> 2013-04-03 10:33:53.541500 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 adding osd.1 to down_pending_out map -23443> 2013-04-03 10:33:53.541556 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 check_osd_map applying osdmap e597 to pg_map -23442> 2013-04-03 10:33:53.541576 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 register_new_pgs checking pg pools for osdmap epoch 597, last_pg_scan 341 -23441> 2013-04-03 10:33:53.541580 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 -23440> 2013-04-03 10:33:53.541585 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -23439> 2013-04-03 10:33:53.541588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -23438> 2013-04-03 10:33:53.541590 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 -23437> 2013-04-03 10:33:53.541593 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 -23436> 2013-04-03 10:33:53.541596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 -23435> 2013-04-03 10:33:53.541598 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 -23434> 2013-04-03 10:33:53.541601 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 -23433> 2013-04-03 10:33:53.541603 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 -23432> 2013-04-03 10:33:53.541619 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs -23431> 2013-04-03 10:33:53.541621 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 check_down_pgs -23430> 2013-04-03 10:33:53.541623 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.b stale with acting [1,0] -23429> 2013-04-03 10:33:53.541626 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 7.6 stale with acting [1,0] -23428> 2013-04-03 10:33:53.541628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 6.7 stale with acting [1,0] -23427> 2013-04-03 10:33:53.541632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.3f stale with acting [1,0] -23426> 2013-04-03 10:33:53.541634 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.3f stale with acting [1,3] -23425> 2013-04-03 10:33:53.541636 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.3c stale with acting [1,3] -23424> 2013-04-03 10:33:53.541639 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.3d stale with acting [1,3] -23423> 2013-04-03 10:33:53.541641 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.3e stale with acting [1,0] -23422> 2013-04-03 10:33:53.541643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.3b stale with acting [1,3] -23421> 2013-04-03 10:33:53.541646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.38 stale with acting [1,4] -23420> 2013-04-03 10:33:53.541648 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.39 stale with acting [1,4] -23419> 2013-04-03 10:33:53.541650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.37 stale with acting [1,4] -23418> 2013-04-03 10:33:53.541652 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.34 stale with acting [1,3] -23417> 2013-04-03 10:33:53.541654 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.35 stale with acting [1,3] -23416> 2013-04-03 10:33:53.541656 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.34 stale with acting [1,2] -23415> 2013-04-03 10:33:53.541658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.33 stale with acting [1,2] -23414> 2013-04-03 10:33:53.541660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.33 stale with acting [1,3] -23413> 2013-04-03 10:33:53.541663 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.30 stale with acting [1,2] -23412> 2013-04-03 10:33:53.541665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.31 stale with acting [1,2] -23411> 2013-04-03 10:33:53.541667 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.32 stale with acting [1,2] -23410> 2013-04-03 10:33:53.541669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.30 stale with acting [1,0] -23409> 2013-04-03 10:33:53.541671 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.2d stale with acting [1,4] -23408> 2013-04-03 10:33:53.541673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.2e stale with acting [1,4] -23407> 2013-04-03 10:33:53.541675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.2f stale with acting [1,4] -23406> 2013-04-03 10:33:53.541677 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.2c stale with acting [1,2] -23405> 2013-04-03 10:33:53.541680 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.2f stale with acting [1,0] -23404> 2013-04-03 10:33:53.541683 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.2e stale with acting [1,2] -23403> 2013-04-03 10:33:53.541685 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.2f stale with acting [1,2] -23402> 2013-04-03 10:33:53.541688 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.2e stale with acting [1,0] -23401> 2013-04-03 10:33:53.541690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.2d stale with acting [1,2] -23400> 2013-04-03 10:33:53.541692 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.25 stale with acting [1,2] -23399> 2013-04-03 10:33:53.541694 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.26 stale with acting [1,2] -23398> 2013-04-03 10:33:53.541696 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.27 stale with acting [1,2] -23397> 2013-04-03 10:33:53.541699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.24 stale with acting [1,3] -23396> 2013-04-03 10:33:53.541701 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.26 stale with acting [1,3] -23395> 2013-04-03 10:33:53.541703 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.25 stale with acting [1,3] -23394> 2013-04-03 10:33:53.541705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.20 stale with acting [1,2] -23393> 2013-04-03 10:33:53.541707 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.1d stale with acting [1,2] -23392> 2013-04-03 10:33:53.541709 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.1e stale with acting [1,2] -23391> 2013-04-03 10:33:53.541712 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.1f stale with acting [1,2] -23390> 2013-04-03 10:33:53.541714 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.1f stale with acting [1,2] -23389> 2013-04-03 10:33:53.541716 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 2.1e stale with acting [1,2] -23388> 2013-04-03 10:33:53.541719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 0.d stale with acting [1,0] -23387> 2013-04-03 10:33:53.541722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 1.c stale with acting [1,0] -23386> 2013-04-03 10:33:53.541724 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 marking pg 8.5 stale with acting [1,0] -23385> 2013-04-03 10:33:53.541770 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 encode_pending v 28959 -23384> 2013-04-03 10:33:53.541891 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427161..427191) queue_proposal bl 17000 bytes; ctx = 0x1b915d0 -23383> 2013-04-03 10:33:53.541897 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427161..427191) propose_queued 427192 17000 bytes -23382> 2013-04-03 10:33:53.544124 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427191 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116780 -23381> 2013-04-03 10:33:53.544157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28958 send_pg_creates to 0 pgs -23380> 2013-04-03 10:33:53.544163 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 send_to_waiting 597 -23379> 2013-04-03 10:33:53.544167 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 share_map_with_random_osd no up osd on our session map -23378> 2013-04-03 10:33:53.544169 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 update_logger -23377> 2013-04-03 10:33:53.544188 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 create_pending e 598 -23376> 2013-04-03 10:33:53.544190 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 remove_redundant_pg_temp -23375> 2013-04-03 10:33:53.544191 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 remove_down_pg_temp -23374> 2013-04-03 10:33:53.544227 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_query osd_boot(osd.1 booted 0 v596) v3 from osd.1 10.252.24.25:6805/3572 -23373> 2013-04-03 10:33:53.544236 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_boot from osd.1 10.252.24.25:6805/3572 -23372> 2013-04-03 10:33:53.544239 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_update osd_boot(osd.1 booted 0 v596) v3 from osd.1 10.252.24.25:6805/3572 -23371> 2013-04-03 10:33:53.544242 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_boot from osd.1 10.252.24.25:6805/3572 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.1 25c71ceb-f510-49a4-bf1a-ba1f96204778 e596 [95,596] lci=[593,596]) cluster_addr 10.252.24.25:6806/3572 hb_addr 10.252.24.25:6807/3572 -23370> 2013-04-03 10:33:53.544254 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 setting osd.1 uuid to 25c71ceb-f510-49a4-bf1a-ba1f96204778 -23369> 2013-04-03 10:33:53.544257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 old osd_info: up_from 593 up_thru 593 down_at 597 last_clean_interval [582,590) -23368> 2013-04-03 10:33:53.544259 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 prepare_boot osd.1 last_clean_interval [582,590) -> [593-596) -23367> 2013-04-03 10:33:53.544263 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 not laggy, new xi down_stamp 2013-04-03 10:33:53.424273 laggy_probability 0 laggy_interval 0 -23366> 2013-04-03 10:33:53.544269 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 should_propose -23365> 2013-04-03 10:33:53.544281 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 update_logger -23364> 2013-04-03 10:33:53.544286 7fcdffcdb700 0 log [INF] : osdmap e597: 5 osds: 4 up, 5 in -23363> 2013-04-03 10:33:53.544287 7fcdffcdb700 10 send_log to self -23362> 2013-04-03 10:33:53.544287 7fcdffcdb700 10 log_queue is 3 last_log 8 sent 7 num 3 unsent 1 sending 1 -23361> 2013-04-03 10:33:53.544289 7fcdffcdb700 10 will send 2013-04-03 10:33:53.494960 mon.0 10.252.24.25:6789/0 7 : [INF] pgmap v28958: 240 pgs: 240 active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 1302B/s wr, 13op/s -23360> 2013-04-03 10:33:53.544294 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x1c15a00 -23359> 2013-04-03 10:33:53.544317 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x1c15a00 con 0x1bac160 -23358> 2013-04-03 10:33:53.544340 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos -23357> 2013-04-03 10:33:53.544354 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos version 393966 summary v 393966 -23356> 2013-04-03 10:33:53.544363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -23355> 2013-04-03 10:33:53.544373 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 preprocess_log log(1 entries) v1 from mon.0 -23354> 2013-04-03 10:33:53.544391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 nothing new -23353> 2013-04-03 10:33:53.551930 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112855 ==== paxos(lease_ack lc 427191 fc 427161 pn 0 opn 0) v3 ==== 80+0+0 (3442845184 0 0) 0x2116780 con 0x208a840 -23352> 2013-04-03 10:33:53.602259 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112856 ==== paxos(accept lc 427191 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2866682499 0 0) 0x20f4500 con 0x208a840 -23351> 2013-04-03 10:33:53.603714 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427192 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116780 -23350> 2013-04-03 10:33:53.603742 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427192 fc 427161 pn 0 opn 0) v3 -- ?+0 0x20f4280 -23349> 2013-04-03 10:33:53.603818 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28958 update_from_paxos applying incremental 28959 -23348> 2013-04-03 10:33:53.603938 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 v28959: 240 pgs: 195 active+clean, 45 stale+active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 0B/s wr, 15op/s -23347> 2013-04-03 10:33:53.603968 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 send_pg_creates to 0 pgs -23346> 2013-04-03 10:33:53.603972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 update_logger -23345> 2013-04-03 10:33:53.603985 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 create_pending v 28960 -23344> 2013-04-03 10:33:53.603994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 check_osd_map already seen 597 >= 597 -23343> 2013-04-03 10:33:53.603997 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 update_logger -23342> 2013-04-03 10:33:53.604008 7fcdffcdb700 0 log [INF] : pgmap v28959: 240 pgs: 195 active+clean, 45 stale+active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 0B/s wr, 15op/s -23341> 2013-04-03 10:33:53.604009 7fcdffcdb700 10 send_log to self -23340> 2013-04-03 10:33:53.604009 7fcdffcdb700 10 log_queue is 4 last_log 9 sent 7 num 4 unsent 2 sending 2 -23339> 2013-04-03 10:33:53.604011 7fcdffcdb700 10 will send 2013-04-03 10:33:53.494960 mon.0 10.252.24.25:6789/0 7 : [INF] pgmap v28958: 240 pgs: 240 active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 1302B/s wr, 13op/s -23338> 2013-04-03 10:33:53.604019 7fcdffcdb700 10 will send 2013-04-03 10:33:53.544287 mon.0 10.252.24.25:6789/0 8 : [INF] osdmap e597: 5 osds: 4 up, 5 in -23337> 2013-04-03 10:33:53.604022 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x21fca00 -23336> 2013-04-03 10:33:53.604063 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos -23335> 2013-04-03 10:33:53.604078 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos version 393966 summary v 393966 -23334> 2013-04-03 10:33:53.604094 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -23333> 2013-04-03 10:33:53.604113 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x21fca00 con 0x1bac160 -23332> 2013-04-03 10:33:53.604134 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos -23331> 2013-04-03 10:33:53.604148 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos version 393966 summary v 393966 -23330> 2013-04-03 10:33:53.604157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23329> 2013-04-03 10:33:53.604169 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 preprocess_log log(2 entries) v1 from mon.0 -23328> 2013-04-03 10:33:53.604192 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23327> 2013-04-03 10:33:53.604204 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 prepare_log log(2 entries) v1 from mon.0 -23326> 2013-04-03 10:33:53.604212 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 logging 2013-04-03 10:33:53.494960 mon.0 10.252.24.25:6789/0 7 : [INF] pgmap v28958: 240 pgs: 240 active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 1302B/s wr, 13op/s -23325> 2013-04-03 10:33:53.604232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 logging 2013-04-03 10:33:53.544287 mon.0 10.252.24.25:6789/0 8 : [INF] osdmap e597: 5 osds: 4 up, 5 in -23324> 2013-04-03 10:33:53.768165 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112857 ==== paxos(lease_ack lc 427192 fc 427161 pn 0 opn 0) v3 ==== 80+0+0 (1740778172 0 0) 0x20f4280 con 0x208a840 -23323> 2013-04-03 10:33:53.768211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos -23322> 2013-04-03 10:33:53.768227 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393966 update_from_paxos version 393966 summary v 393966 -23321> 2013-04-03 10:33:53.768241 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -23320> 2013-04-03 10:33:53.768258 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112858 ==== forward(osd_alive(want up_thru 597 have 597) v1) to leader v1 ==== 294+0+0 (3486498143 0 0) 0x1bae340 con 0x208a840 -23319> 2013-04-03 10:33:53.768266 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -23318> 2013-04-03 10:33:53.768271 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2294a80 from 10.252.24.88:6789/0 -23317> 2013-04-03 10:33:53.768292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_query osd_alive(want up_thru 597 have 597) v1 from osd.0 10.252.58.31:6802/2682 -23316> 2013-04-03 10:33:53.768297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_alive want up_thru 597 from osd.0 10.252.58.31:6802/2682 -23315> 2013-04-03 10:33:53.768301 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_update osd_alive(want up_thru 597 have 597) v1 from osd.0 10.252.58.31:6802/2682 -23314> 2013-04-03 10:33:53.768306 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_alive want up_thru 597 have 597 from osd.0 10.252.58.31:6802/2682 -23313> 2013-04-03 10:33:53.768310 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 should_propose -23312> 2013-04-03 10:33:53.770061 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112859 ==== forward(pg_stats(30 pgs tid 30429 v 597) v1) to leader v1 ==== 11478+0+0 (729369230 0 0) 0x25f0580 con 0x208a840 -23311> 2013-04-03 10:33:53.770072 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -23310> 2013-04-03 10:33:53.770080 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1c18900 from 10.252.24.88:6789/0 -23309> 2013-04-03 10:33:53.770104 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 preprocess_query pg_stats(30 pgs tid 30429 v 597) v1 from osd.3 10.252.24.88:6801/17538 -23308> 2013-04-03 10:33:53.770111 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 prepare_update pg_stats(30 pgs tid 30429 v 597) v1 from osd.3 10.252.24.88:6801/17538 -23307> 2013-04-03 10:33:53.770115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 prepare_pg_stats pg_stats(30 pgs tid 30429 v 597) v1 from osd.3 -23306> 2013-04-03 10:33:53.770119 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 got osd.3 osd_stat(1187 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1187 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -23305> 2013-04-03 10:33:53.770164 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112860 ==== forward(osd_alive(want up_thru 597 have 597) v1) to leader v1 ==== 294+0+0 (1881033317 0 0) 0x25f02c0 con 0x208a840 -23304> 2013-04-03 10:33:53.770174 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -23303> 2013-04-03 10:33:53.770181 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2294c40 from 10.252.24.88:6789/0 -23302> 2013-04-03 10:33:53.770203 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_query osd_alive(want up_thru 597 have 597) v1 from osd.3 10.252.24.88:6801/17538 -23301> 2013-04-03 10:33:53.770209 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_alive want up_thru 597 from osd.3 10.252.24.88:6801/17538 -23300> 2013-04-03 10:33:53.770212 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_update osd_alive(want up_thru 597 have 597) v1 from osd.3 10.252.24.88:6801/17538 -23299> 2013-04-03 10:33:53.770215 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_alive want up_thru 597 have 597 from osd.3 10.252.24.88:6801/17538 -23298> 2013-04-03 10:33:53.770218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 should_propose -23297> 2013-04-03 10:33:54.351358 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112861 ==== forward(osd_alive(want up_thru 597 have 597) v1) to leader v1 ==== 294+0+0 (215695374 0 0) 0x25f0000 con 0x208a840 -23296> 2013-04-03 10:33:54.351383 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 -23295> 2013-04-03 10:33:54.351393 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2294e00 from 10.252.24.88:6789/0 -23294> 2013-04-03 10:33:54.351440 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_query osd_alive(want up_thru 597 have 597) v1 from osd.4 10.252.24.32:6800/13473 -23293> 2013-04-03 10:33:54.351448 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 preprocess_alive want up_thru 597 from osd.4 10.252.24.32:6800/13473 -23292> 2013-04-03 10:33:54.351452 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_update osd_alive(want up_thru 597 have 597) v1 from osd.4 10.252.24.32:6800/13473 -23291> 2013-04-03 10:33:54.351458 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 prepare_alive want up_thru 597 have 597 from osd.4 10.252.24.32:6800/13473 -23290> 2013-04-03 10:33:54.351462 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e597 should_propose -23289> 2013-04-03 10:33:54.413325 7fcdfead7700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2116c80 sd=23 :6789 s=0 pgs=0 cs=0 l=0).accept sd=23 10.252.24.88:32935/0 -23288> 2013-04-03 10:33:54.413641 7fcdfead7700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.88:6800/17213 mds protocol 0 -23287> 2013-04-03 10:33:54.413978 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 1 ==== auth(proto 0 36 bytes epoch 5) v1 ==== 66+0+0 (2713832624 0 0) 0x25f0000 con 0x200d420 -23286> 2013-04-03 10:33:54.413997 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -23285> 2013-04-03 10:33:54.414004 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: mds.? 10.252.24.88:6800/17213 is open for mds.? 10.252.24.88:6800/17213 -23284> 2013-04-03 10:33:54.414018 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -23283> 2013-04-03 10:33:54.414043 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -23282> 2013-04-03 10:33:54.414059 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 preprocess_query auth(proto 0 36 bytes epoch 5) v1 from mds.? 10.252.24.88:6800/17213 -23281> 2013-04-03 10:33:54.414071 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 prep_auth() blob_size=36 -23280> 2013-04-03 10:33:54.414103 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x21fee00 con 0x200d420 -23279> 2013-04-03 10:33:54.444038 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (3672189860 0 0) 0x25f0400 con 0x200d420 -23278> 2013-04-03 10:33:54.444079 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -23277> 2013-04-03 10:33:54.444096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 preprocess_query auth(proto 2 144 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/17213 -23276> 2013-04-03 10:33:54.444107 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 prep_auth() blob_size=144 -23275> 2013-04-03 10:33:54.444958 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x25f0000 con 0x200d420 -23274> 2013-04-03 10:33:54.445572 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 3 ==== mon_subscribe({mdsmap=185+,monmap=6+}) v2 ==== 42+0+0 (945815558 0 0) 0x2294700 con 0x200d420 -23273> 2013-04-03 10:33:54.445593 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=185+,monmap=6+}) v2 -23272> 2013-04-03 10:33:54.445608 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -23271> 2013-04-03 10:33:54.445610 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/17213 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2224000 -23270> 2013-04-03 10:33:54.493440 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393966 encode_pending v393967 -23269> 2013-04-03 10:33:54.493465 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427161..427192) queue_proposal bl 946 bytes; ctx = 0x1b91520 -23268> 2013-04-03 10:33:54.493473 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427161..427192) propose_queued 427193 946 bytes -23267> 2013-04-03 10:33:54.494908 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427192 fc 0 pn 17600 opn 0) v3 -- ?+0 0x1f91c80 -23266> 2013-04-03 10:33:54.526207 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112862 ==== paxos(accept lc 427192 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3032015158 0 0) 0x1f91c80 con 0x208a840 -23265> 2013-04-03 10:33:54.527062 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427193 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4280 -23264> 2013-04-03 10:33:54.527095 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427193 fc 427161 pn 0 opn 0) v3 -- ?+0 0x20f4500 -23263> 2013-04-03 10:33:54.527132 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos -23262> 2013-04-03 10:33:54.527155 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393966 -23261> 2013-04-03 10:33:54.527168 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos latest full 393798 -23260> 2013-04-03 10:33:54.527186 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393967 update_from_paxos applying incremental log 393967 2013-04-03 10:33:53.373245 mon.0 10.252.24.25:6789/0 6 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -23259> 2013-04-03 10:33:54.527213 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393967 update_from_paxos applying incremental log 393967 2013-04-03 10:33:53.494960 mon.0 10.252.24.25:6789/0 7 : [INF] pgmap v28958: 240 pgs: 240 active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 1302B/s wr, 13op/s -23258> 2013-04-03 10:33:54.527234 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393967 update_from_paxos applying incremental log 393967 2013-04-03 10:33:53.544287 mon.0 10.252.24.25:6789/0 8 : [INF] osdmap e597: 5 osds: 4 up, 5 in -23257> 2013-04-03 10:33:54.527268 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 check_subs -23256> 2013-04-03 10:33:54.527295 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 create_pending v 393968 -23255> 2013-04-03 10:33:54.527316 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393967 _updated_log for mon.0 10.252.24.25:6789/0 -23254> 2013-04-03 10:33:54.527328 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 7) v1 -- ?+0 0x2294700 con 0x1bac160 -23253> 2013-04-03 10:33:54.527335 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393967 _updated_log for mon.0 10.252.24.25:6789/0 -23252> 2013-04-03 10:33:54.527347 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 8) v1 -- ?+0 0x22241c0 con 0x1bac160 -23251> 2013-04-03 10:33:54.527388 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos -23250> 2013-04-03 10:33:54.527405 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393967 -23249> 2013-04-03 10:33:54.527421 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -23248> 2013-04-03 10:33:54.527439 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 7) v1 ==== 0+0+0 (0 0 0) 0x2294700 con 0x1bac160 -23247> 2013-04-03 10:33:54.527447 7fcdffcdb700 10 handle_log_ack log(last 7) v1 -23246> 2013-04-03 10:33:54.527448 7fcdffcdb700 10 logged 2013-04-03 10:33:53.373245 mon.0 10.252.24.25:6789/0 6 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -23245> 2013-04-03 10:33:54.527454 7fcdffcdb700 10 logged 2013-04-03 10:33:53.494960 mon.0 10.252.24.25:6789/0 7 : [INF] pgmap v28958: 240 pgs: 240 active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 1302B/s wr, 13op/s -23244> 2013-04-03 10:33:54.527459 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 8) v1 ==== 0+0+0 (0 0 0) 0x22241c0 con 0x1bac160 -23243> 2013-04-03 10:33:54.527464 7fcdffcdb700 10 handle_log_ack log(last 8) v1 -23242> 2013-04-03 10:33:54.527465 7fcdffcdb700 10 logged 2013-04-03 10:33:53.544287 mon.0 10.252.24.25:6789/0 8 : [INF] osdmap e597: 5 osds: 4 up, 5 in -23241> 2013-04-03 10:33:54.538167 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112863 ==== paxos(lease_ack lc 427193 fc 427161 pn 0 opn 0) v3 ==== 80+0+0 (2510528612 0 0) 0x20f4500 con 0x208a840 -23240> 2013-04-03 10:33:54.538205 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos -23239> 2013-04-03 10:33:54.538221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393967 -23238> 2013-04-03 10:33:54.538237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -23237> 2013-04-03 10:33:54.540048 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e597 min_last_epoch_clean 593 -23236> 2013-04-03 10:33:54.540108 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e597 min_last_epoch_clean 593 -23235> 2013-04-03 10:33:54.540162 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e597 encode_pending e 598 -23234> 2013-04-03 10:33:54.540166 7fce004dc700 2 mon.hotcpc9039@0(leader).osd e597 osd.1 UP 10.252.24.25:6805/3572 -23233> 2013-04-03 10:33:54.540193 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427161..427193) queue_proposal bl 793 bytes; ctx = 0x1b915f0 -23232> 2013-04-03 10:33:54.540199 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427161..427193) propose_queued 427194 793 bytes -23231> 2013-04-03 10:33:54.540603 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427193 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041500 -23230> 2013-04-03 10:33:54.551327 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112864 ==== paxos(accept lc 427193 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4081088180 0 0) 0x2041500 con 0x208a840 -23229> 2013-04-03 10:33:54.558472 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427194 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4500 -23228> 2013-04-03 10:33:54.558507 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427194 fc 427161 pn 0 opn 0) v3 -- ?+0 0x1f91c80 -23227> 2013-04-03 10:33:54.558575 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e597 update_from_paxos applying incremental 598 -23226> 2013-04-03 10:33:54.558625 7fcdffcdb700 1 mon.hotcpc9039@0(leader).osd e598 e598: 5 osds: 5 up, 5 in -23225> 2013-04-03 10:33:54.559927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 check_osd_map applying osdmap e598 to pg_map -23224> 2013-04-03 10:33:54.559952 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 register_new_pgs checking pg pools for osdmap epoch 598, last_pg_scan 341 -23223> 2013-04-03 10:33:54.559957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 -23222> 2013-04-03 10:33:54.559961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -23221> 2013-04-03 10:33:54.559963 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -23220> 2013-04-03 10:33:54.559966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 -23219> 2013-04-03 10:33:54.559968 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 -23218> 2013-04-03 10:33:54.559971 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 -23217> 2013-04-03 10:33:54.559973 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 -23216> 2013-04-03 10:33:54.559975 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 -23215> 2013-04-03 10:33:54.559979 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 -23214> 2013-04-03 10:33:54.559988 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs -23213> 2013-04-03 10:33:54.559990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 check_down_pgs -23212> 2013-04-03 10:33:54.560039 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 encode_pending v 28960 -23211> 2013-04-03 10:33:54.560146 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427161..427194) queue_proposal bl 11375 bytes; ctx = 0x1b914b0 -23210> 2013-04-03 10:33:54.560153 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427161..427194) propose_queued 427195 11375 bytes -23209> 2013-04-03 10:33:54.561163 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427194 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116500 -23208> 2013-04-03 10:33:54.561199 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28959 send_pg_creates to 0 pgs -23207> 2013-04-03 10:33:54.561203 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 send_to_waiting 598 -23206> 2013-04-03 10:33:54.561206 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 share_map_with_random_osd no up osd on our session map -23205> 2013-04-03 10:33:54.561209 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 update_logger -23204> 2013-04-03 10:33:54.561228 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 create_pending e 599 -23203> 2013-04-03 10:33:54.561230 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 remove_redundant_pg_temp -23202> 2013-04-03 10:33:54.561232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 remove_down_pg_temp -23201> 2013-04-03 10:33:54.561251 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 _booted osd.1 10.252.24.25:6805/3572 w 0 from 596 -23200> 2013-04-03 10:33:54.561268 7fcdffcdb700 0 log [INF] : osd.1 10.252.24.25:6805/3572 boot -23199> 2013-04-03 10:33:54.561270 7fcdffcdb700 10 send_log to self -23198> 2013-04-03 10:33:54.561270 7fcdffcdb700 10 log_queue is 2 last_log 10 sent 8 num 2 unsent 2 sending 2 -23197> 2013-04-03 10:33:54.561272 7fcdffcdb700 10 will send 2013-04-03 10:33:53.604009 mon.0 10.252.24.25:6789/0 9 : [INF] pgmap v28959: 240 pgs: 195 active+clean, 45 stale+active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 0B/s wr, 15op/s -23196> 2013-04-03 10:33:54.561278 7fcdffcdb700 10 will send 2013-04-03 10:33:54.561269 mon.0 10.252.24.25:6789/0 10 : [INF] osd.1 10.252.24.25:6805/3572 boot -23195> 2013-04-03 10:33:54.561282 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x21fca00 -23194> 2013-04-03 10:33:54.561301 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_latest to osd.1 10.252.24.25:6805/3572 start 597 -23193> 2013-04-03 10:33:54.561307 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_incremental [597..598] to osd.1 10.252.24.25:6805/3572 -23192> 2013-04-03 10:33:54.561331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 build_incremental [597..598] -23191> 2013-04-03 10:33:54.561367 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(597..598 src has 97..598) v3 tid 17358) v2 -- ?+0 0x20add80 con 0x208a840 -23190> 2013-04-03 10:33:54.561382 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 _reply_map 597 from osd.0 10.252.58.31:6802/2682 -23189> 2013-04-03 10:33:54.561402 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_latest to osd.0 10.252.58.31:6802/2682 start 597 -23188> 2013-04-03 10:33:54.561407 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_incremental [597..598] to osd.0 10.252.58.31:6802/2682 -23187> 2013-04-03 10:33:54.561417 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 build_incremental [597..598] -23186> 2013-04-03 10:33:54.561446 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(597..598 src has 97..598) v3 tid 17365) v2 -- ?+0 0x20ad6c0 con 0x208a840 -23185> 2013-04-03 10:33:54.561456 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 _reply_map 597 from osd.3 10.252.24.88:6801/17538 -23184> 2013-04-03 10:33:54.561474 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_latest to osd.3 10.252.24.88:6801/17538 start 597 -23183> 2013-04-03 10:33:54.561479 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_incremental [597..598] to osd.3 10.252.24.88:6801/17538 -23182> 2013-04-03 10:33:54.561489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 build_incremental [597..598] -23181> 2013-04-03 10:33:54.561519 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(597..598 src has 97..598) v3 tid 17367) v2 -- ?+0 0x20ad240 con 0x208a840 -23180> 2013-04-03 10:33:54.561529 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 _reply_map 597 from osd.4 10.252.24.32:6800/13473 -23179> 2013-04-03 10:33:54.561547 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_latest to osd.4 10.252.24.32:6800/13473 start 597 -23178> 2013-04-03 10:33:54.561552 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e598 send_incremental [597..598] to osd.4 10.252.24.32:6800/13473 -23177> 2013-04-03 10:33:54.561568 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 build_incremental [597..598] -23176> 2013-04-03 10:33:54.561597 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(597..598 src has 97..598) v3 tid 17368) v2 -- ?+0 0x20fc000 con 0x208a840 -23175> 2013-04-03 10:33:54.561607 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 update_logger -23174> 2013-04-03 10:33:54.561613 7fcdffcdb700 0 log [INF] : osdmap e598: 5 osds: 5 up, 5 in -23173> 2013-04-03 10:33:54.561614 7fcdffcdb700 10 send_log to self -23172> 2013-04-03 10:33:54.561614 7fcdffcdb700 10 log_queue is 3 last_log 11 sent 10 num 3 unsent 1 sending 1 -23171> 2013-04-03 10:33:54.561616 7fcdffcdb700 10 will send 2013-04-03 10:33:54.561269 mon.0 10.252.24.25:6789/0 10 : [INF] osd.1 10.252.24.25:6805/3572 boot -23170> 2013-04-03 10:33:54.561622 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x21fea00 -23169> 2013-04-03 10:33:54.561644 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x21fca00 con 0x1bac160 -23168> 2013-04-03 10:33:54.561665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos -23167> 2013-04-03 10:33:54.561680 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393967 -23166> 2013-04-03 10:33:54.561689 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23165> 2013-04-03 10:33:54.561699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 preprocess_log log(2 entries) v1 from mon.0 -23164> 2013-04-03 10:33:54.561723 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23163> 2013-04-03 10:33:54.561735 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 prepare_log log(2 entries) v1 from mon.0 -23162> 2013-04-03 10:33:54.561744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 logging 2013-04-03 10:33:53.604009 mon.0 10.252.24.25:6789/0 9 : [INF] pgmap v28959: 240 pgs: 195 active+clean, 45 stale+active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 0B/s wr, 15op/s -23161> 2013-04-03 10:33:54.561765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 logging 2013-04-03 10:33:54.561269 mon.0 10.252.24.25:6789/0 10 : [INF] osd.1 10.252.24.25:6805/3572 boot -23160> 2013-04-03 10:33:54.561796 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x21fea00 con 0x1bac160 -23159> 2013-04-03 10:33:54.561815 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos -23158> 2013-04-03 10:33:54.561831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393967 -23157> 2013-04-03 10:33:54.561840 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -23156> 2013-04-03 10:33:54.561850 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 preprocess_log log(1 entries) v1 from mon.0 -23155> 2013-04-03 10:33:54.561866 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 nothing new -23154> 2013-04-03 10:33:54.584959 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112865 ==== paxos(lease_ack lc 427194 fc 427161 pn 0 opn 0) v3 ==== 80+0+0 (141935639 0 0) 0x2041a00 con 0x208a840 -23153> 2013-04-03 10:33:54.634092 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112866 ==== paxos(accept lc 427194 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2441357525 0 0) 0x2041c80 con 0x208a840 -23152> 2013-04-03 10:33:54.672843 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427195 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041a00 -23151> 2013-04-03 10:33:54.672864 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427195 fc 427161 pn 0 opn 0) v3 -- ?+0 0x2041500 -23150> 2013-04-03 10:33:54.672946 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28959 update_from_paxos applying incremental 28960 -23149> 2013-04-03 10:33:54.673036 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 v28960: 240 pgs: 190 active+clean, 38 stale+active+clean, 12 peering; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail -23148> 2013-04-03 10:33:54.673062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 send_pg_creates to 0 pgs -23147> 2013-04-03 10:33:54.673065 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 update_logger -23146> 2013-04-03 10:33:54.673077 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 create_pending v 28961 -23145> 2013-04-03 10:33:54.673085 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28960 _updated_stats for osd.3 10.252.24.88:6801/17538 -23144> 2013-04-03 10:33:54.673091 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(30 pgs tid 30429) v1 tid 17366) v2 -- ?+0 0x20fcb40 con 0x208a840 -23143> 2013-04-03 10:33:54.673106 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 check_osd_map already seen 598 >= 598 -23142> 2013-04-03 10:33:54.673110 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 update_logger -23141> 2013-04-03 10:33:54.673121 7fcdffcdb700 0 log [INF] : pgmap v28960: 240 pgs: 190 active+clean, 38 stale+active+clean, 12 peering; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail -23140> 2013-04-03 10:33:54.673122 7fcdffcdb700 10 send_log to self -23139> 2013-04-03 10:33:54.673124 7fcdffcdb700 10 log_queue is 4 last_log 12 sent 10 num 4 unsent 2 sending 2 -23138> 2013-04-03 10:33:54.673125 7fcdffcdb700 10 will send 2013-04-03 10:33:54.561269 mon.0 10.252.24.25:6789/0 10 : [INF] osd.1 10.252.24.25:6805/3572 boot -23137> 2013-04-03 10:33:54.673131 7fcdffcdb700 10 will send 2013-04-03 10:33:54.561614 mon.0 10.252.24.25:6789/0 11 : [INF] osdmap e598: 5 osds: 5 up, 5 in -23136> 2013-04-03 10:33:54.673136 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2201000 -23135> 2013-04-03 10:33:54.673179 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos -23134> 2013-04-03 10:33:54.673195 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393967 -23133> 2013-04-03 10:33:54.673212 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -23132> 2013-04-03 10:33:54.673231 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2201000 con 0x1bac160 -23131> 2013-04-03 10:33:54.673251 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos -23130> 2013-04-03 10:33:54.673266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393967 -23129> 2013-04-03 10:33:54.673274 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23128> 2013-04-03 10:33:54.673286 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 preprocess_log log(2 entries) v1 from mon.0 -23127> 2013-04-03 10:33:54.673318 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -23126> 2013-04-03 10:33:54.673332 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 prepare_log log(2 entries) v1 from mon.0 -23125> 2013-04-03 10:33:54.673343 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 logging 2013-04-03 10:33:54.561269 mon.0 10.252.24.25:6789/0 10 : [INF] osd.1 10.252.24.25:6805/3572 boot -23124> 2013-04-03 10:33:54.673362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 logging 2013-04-03 10:33:54.561614 mon.0 10.252.24.25:6789/0 11 : [INF] osdmap e598: 5 osds: 5 up, 5 in -23123> 2013-04-03 10:33:54.892784 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112867 ==== paxos(lease_ack lc 427195 fc 427161 pn 0 opn 0) v3 ==== 80+0+0 (3209261389 0 0) 0x2041500 con 0x208a840 -23122> 2013-04-03 10:33:54.892852 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos -23121> 2013-04-03 10:33:54.892870 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393967 update_from_paxos version 393967 summary v 393967 -23120> 2013-04-03 10:33:54.892888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -23119> 2013-04-03 10:33:54.892908 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112868 ==== forward(osd_alive(want up_thru 598 have 598) v1) to leader v1 ==== 294+0+0 (639776935 0 0) 0x1bae8c0 con 0x208a840 -23118> 2013-04-03 10:33:54.892918 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -23117> 2013-04-03 10:33:54.892926 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x22248c0 from 10.252.24.88:6789/0 -23116> 2013-04-03 10:33:54.892949 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_query osd_alive(want up_thru 598 have 598) v1 from osd.1 10.252.24.25:6805/3572 -23115> 2013-04-03 10:33:54.892956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_alive want up_thru 598 from osd.1 10.252.24.25:6805/3572 -23114> 2013-04-03 10:33:54.892960 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_update osd_alive(want up_thru 598 have 598) v1 from osd.1 10.252.24.25:6805/3572 -23113> 2013-04-03 10:33:54.892964 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_alive want up_thru 598 have 598 from osd.1 10.252.24.25:6805/3572 -23112> 2013-04-03 10:33:54.892967 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 should_propose -23111> 2013-04-03 10:33:54.893171 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112869 ==== forward(osd_alive(want up_thru 598 have 598) v1) to leader v1 ==== 294+0+0 (672993898 0 0) 0x1bae080 con 0x208a840 -23110> 2013-04-03 10:33:54.893187 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 -23109> 2013-04-03 10:33:54.893194 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x22941c0 from 10.252.24.88:6789/0 -23108> 2013-04-03 10:33:54.893216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_query osd_alive(want up_thru 598 have 598) v1 from osd.2 10.252.24.30:6801/10373 -23107> 2013-04-03 10:33:54.893223 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_alive want up_thru 598 from osd.2 10.252.24.30:6801/10373 -23106> 2013-04-03 10:33:54.893226 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_update osd_alive(want up_thru 598 have 598) v1 from osd.2 10.252.24.30:6801/10373 -23105> 2013-04-03 10:33:54.893229 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_alive want up_thru 598 have 598 from osd.2 10.252.24.30:6801/10373 -23104> 2013-04-03 10:33:54.893232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 should_propose -23103> 2013-04-03 10:33:54.893243 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112870 ==== forward(osd_alive(want up_thru 598 have 598) v1) to leader v1 ==== 294+0+0 (2427565252 0 0) 0x1baeb80 con 0x208a840 -23102> 2013-04-03 10:33:54.893249 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -23101> 2013-04-03 10:33:54.893255 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2294380 from 10.252.24.88:6789/0 -23100> 2013-04-03 10:33:54.893275 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_query osd_alive(want up_thru 598 have 598) v1 from osd.0 10.252.58.31:6802/2682 -23099> 2013-04-03 10:33:54.893281 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_alive want up_thru 598 from osd.0 10.252.58.31:6802/2682 -23098> 2013-04-03 10:33:54.893286 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_update osd_alive(want up_thru 598 have 598) v1 from osd.0 10.252.58.31:6802/2682 -23097> 2013-04-03 10:33:54.893289 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_alive want up_thru 598 have 598 from osd.0 10.252.58.31:6802/2682 -23096> 2013-04-03 10:33:54.893291 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 should_propose -23095> 2013-04-03 10:33:54.893452 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112871 ==== forward(mdsbeacon(11234/hotcpc9039 up:replay seq 4 v184) v2) to leader v1 ==== 560+0+0 (3119518282 0 0) 0x1bae600 con 0x208a840 -23094> 2013-04-03 10:33:54.893462 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -23093> 2013-04-03 10:33:54.893468 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae340 from 10.252.24.88:6789/0 -23092> 2013-04-03 10:33:54.893491 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 4 v184) v2 from mds.0 10.252.24.25:6802/3144 -23091> 2013-04-03 10:33:54.893502 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:replay seq 4 v184) v2 tid 17372) v2 -- ?+0 0x1c18900 con 0x208a840 -23090> 2013-04-03 10:33:54.936869 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112872 ==== forward(osd_alive(want up_thru 598 have 598) v1) to leader v1 ==== 294+0+0 (4095220933 0 0) 0x1baeb80 con 0x208a840 -23089> 2013-04-03 10:33:54.936884 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -23088> 2013-04-03 10:33:54.936891 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2294540 from 10.252.24.88:6789/0 -23087> 2013-04-03 10:33:54.936923 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_query osd_alive(want up_thru 598 have 598) v1 from osd.3 10.252.24.88:6801/17538 -23086> 2013-04-03 10:33:54.936930 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_alive want up_thru 598 from osd.3 10.252.24.88:6801/17538 -23085> 2013-04-03 10:33:54.936933 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_update osd_alive(want up_thru 598 have 598) v1 from osd.3 10.252.24.88:6801/17538 -23084> 2013-04-03 10:33:54.936936 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_alive want up_thru 598 have 598 from osd.3 10.252.24.88:6801/17538 -23083> 2013-04-03 10:33:54.936939 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 should_propose -23082> 2013-04-03 10:33:55.550247 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112873 ==== forward(osd_alive(want up_thru 598 have 598) v1) to leader v1 ==== 294+0+0 (3879944819 0 0) 0x20e4340 con 0x208a840 -23081> 2013-04-03 10:33:55.550269 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -23080> 2013-04-03 10:33:55.550278 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2224380 from 10.252.24.88:6789/0 -23079> 2013-04-03 10:33:55.550326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_query osd_alive(want up_thru 598 have 598) v1 from osd.1 10.252.24.25:6805/3572 -23078> 2013-04-03 10:33:55.550334 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 preprocess_alive want up_thru 598 from osd.1 10.252.24.25:6805/3572 -23077> 2013-04-03 10:33:55.550338 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_update osd_alive(want up_thru 598 have 598) v1 from osd.1 10.252.24.25:6805/3572 -23076> 2013-04-03 10:33:55.550341 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 prepare_alive want up_thru 598 have 598 from osd.1 10.252.24.25:6805/3572 -23075> 2013-04-03 10:33:55.550345 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e598 should_propose -23074> 2013-04-03 10:33:55.551510 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393967 encode_pending v393968 -23073> 2013-04-03 10:33:55.551536 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427161..427195) queue_proposal bl 883 bytes; ctx = 0x1b915c0 -23072> 2013-04-03 10:33:55.551545 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427161..427195) propose_queued 427196 883 bytes -23071> 2013-04-03 10:33:55.552613 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427195 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041280 -23070> 2013-04-03 10:33:55.553930 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112874 ==== forward(pg_stats(45 pgs tid 1 v 598) v1) to leader v1 ==== 17153+0+0 (1714515088 0 0) 0x20e4080 con 0x208a840 -23069> 2013-04-03 10:33:55.553941 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -23068> 2013-04-03 10:33:55.553948 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1c18900 from 10.252.24.88:6789/0 -23067> 2013-04-03 10:33:55.553975 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 preprocess_query pg_stats(45 pgs tid 1 v 598) v1 from osd.1 10.252.24.25:6805/3572 -23066> 2013-04-03 10:33:55.553982 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 prepare_update pg_stats(45 pgs tid 1 v 598) v1 from osd.1 10.252.24.25:6805/3572 -23065> 2013-04-03 10:33:55.553986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 prepare_pg_stats pg_stats(45 pgs tid 1 v 598) v1 from osd.1 -23064> 2013-04-03 10:33:55.553992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 got osd.1 osd_stat(1110 MB used, 148 GB avail, 149 GB total, peers []/[]) (was osd_stat(1112 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -23063> 2013-04-03 10:33:55.574878 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112875 ==== paxos(accept lc 427195 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (375291141 0 0) 0x2041280 con 0x208a840 -23062> 2013-04-03 10:33:55.575922 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427196 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041500 -23061> 2013-04-03 10:33:55.575954 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427196 fc 427161 pn 0 opn 0) v3 -- ?+0 0x2041c80 -23060> 2013-04-03 10:33:55.575996 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos -23059> 2013-04-03 10:33:55.576022 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos version 393968 summary v 393967 -23058> 2013-04-03 10:33:55.576036 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos latest full 393798 -23057> 2013-04-03 10:33:55.576053 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393968 update_from_paxos applying incremental log 393968 2013-04-03 10:33:53.604009 mon.0 10.252.24.25:6789/0 9 : [INF] pgmap v28959: 240 pgs: 195 active+clean, 45 stale+active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 0B/s wr, 15op/s -23056> 2013-04-03 10:33:55.576077 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393968 update_from_paxos applying incremental log 393968 2013-04-03 10:33:54.561269 mon.0 10.252.24.25:6789/0 10 : [INF] osd.1 10.252.24.25:6805/3572 boot -23055> 2013-04-03 10:33:55.576096 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393968 update_from_paxos applying incremental log 393968 2013-04-03 10:33:54.561614 mon.0 10.252.24.25:6789/0 11 : [INF] osdmap e598: 5 osds: 5 up, 5 in -23054> 2013-04-03 10:33:55.576135 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 check_subs -23053> 2013-04-03 10:33:55.576162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 create_pending v 393969 -23052> 2013-04-03 10:33:55.576183 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393968 _updated_log for mon.0 10.252.24.25:6789/0 -23051> 2013-04-03 10:33:55.576195 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 10) v1 -- ?+0 0x2294c40 con 0x1bac160 -23050> 2013-04-03 10:33:55.576203 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393968 _updated_log for mon.0 10.252.24.25:6789/0 -23049> 2013-04-03 10:33:55.576215 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 11) v1 -- ?+0 0x2294a80 con 0x1bac160 -23048> 2013-04-03 10:33:55.576255 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos -23047> 2013-04-03 10:33:55.576269 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos version 393968 summary v 393968 -23046> 2013-04-03 10:33:55.576286 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -23045> 2013-04-03 10:33:55.576305 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 10) v1 ==== 0+0+0 (0 0 0) 0x2294c40 con 0x1bac160 -23044> 2013-04-03 10:33:55.576313 7fcdffcdb700 10 handle_log_ack log(last 10) v1 -23043> 2013-04-03 10:33:55.576316 7fcdffcdb700 10 logged 2013-04-03 10:33:53.604009 mon.0 10.252.24.25:6789/0 9 : [INF] pgmap v28959: 240 pgs: 195 active+clean, 45 stale+active+clean; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail; 0B/s rd, 0B/s wr, 15op/s -23042> 2013-04-03 10:33:55.576323 7fcdffcdb700 10 logged 2013-04-03 10:33:54.561269 mon.0 10.252.24.25:6789/0 10 : [INF] osd.1 10.252.24.25:6805/3572 boot -23041> 2013-04-03 10:33:55.576328 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 11) v1 ==== 0+0+0 (0 0 0) 0x2294a80 con 0x1bac160 -23040> 2013-04-03 10:33:55.576335 7fcdffcdb700 10 handle_log_ack log(last 11) v1 -23039> 2013-04-03 10:33:55.576336 7fcdffcdb700 10 logged 2013-04-03 10:33:54.561614 mon.0 10.252.24.25:6789/0 11 : [INF] osdmap e598: 5 osds: 5 up, 5 in -23038> 2013-04-03 10:33:55.576342 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 4 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140292 v184) v2 ==== 289+0+0 (471207938 0 0) 0x1bae8c0 con 0x2118420 -23037> 2013-04-03 10:33:55.576370 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140292 v184) v2 from mds.? 10.252.58.31:6800/10908 -23036> 2013-04-03 10:33:55.576379 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140292 v184) v2 -- ?+0 0x20e4080 con 0x2118420 -23035> 2013-04-03 10:33:55.588172 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112876 ==== paxos(lease_ack lc 427196 fc 427161 pn 0 opn 0) v3 ==== 80+0+0 (2472974394 0 0) 0x2041c80 con 0x208a840 -23034> 2013-04-03 10:33:55.588205 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos -23033> 2013-04-03 10:33:55.588221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos version 393968 summary v 393968 -23032> 2013-04-03 10:33:55.588237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -23031> 2013-04-03 10:33:55.634223 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e598 min_last_epoch_clean 593 -23030> 2013-04-03 10:33:55.634292 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e598 min_last_epoch_clean 593 -23029> 2013-04-03 10:33:55.634341 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e598 encode_pending e 599 -23028> 2013-04-03 10:33:55.634374 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427161..427196) queue_proposal bl 316 bytes; ctx = 0x1b914d0 -23027> 2013-04-03 10:33:55.634383 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427161..427196) propose_queued 427197 316 bytes -23026> 2013-04-03 10:33:55.634851 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427196 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041000 -23025> 2013-04-03 10:33:55.634946 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28960 encode_pending v 28961 -23024> 2013-04-03 10:33:55.635106 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427161..427196) queue_proposal bl 17050 bytes; ctx = 0x1b91490 -23023> 2013-04-03 10:33:55.635112 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427161..427196) propose_new_value not active; proposal queued -23022> 2013-04-03 10:33:55.650072 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112877 ==== paxos(accept lc 427196 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2735792121 0 0) 0x2041000 con 0x208a840 -23021> 2013-04-03 10:33:55.650425 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427197 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041c80 -23020> 2013-04-03 10:33:55.650458 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427197 fc 427161 pn 0 opn 0) v3 -- ?+0 0x2041280 -23019> 2013-04-03 10:33:55.650533 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e598 update_from_paxos applying incremental 599 -23018> 2013-04-03 10:33:55.650576 7fcdffcdb700 1 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in -23017> 2013-04-03 10:33:55.651761 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28960 check_osd_map -- pgmap not writeable, waiting -23016> 2013-04-03 10:33:55.651769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 send_to_waiting 599 -23015> 2013-04-03 10:33:55.651773 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 share_map_with_random_osd no up osd on our session map -23014> 2013-04-03 10:33:55.651774 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 update_logger -23013> 2013-04-03 10:33:55.651790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 create_pending e 600 -23012> 2013-04-03 10:33:55.651793 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 remove_redundant_pg_temp -23011> 2013-04-03 10:33:55.651794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 remove_down_pg_temp -23010> 2013-04-03 10:33:55.651813 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e599 _reply_map 598 from osd.1 10.252.24.25:6805/3572 -23009> 2013-04-03 10:33:55.651831 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_latest to osd.1 10.252.24.25:6805/3572 start 598 -23008> 2013-04-03 10:33:55.651834 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_incremental [598..599] to osd.1 10.252.24.25:6805/3572 -23007> 2013-04-03 10:33:55.651845 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 build_incremental [598..599] -23006> 2013-04-03 10:33:55.651876 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(598..599 src has 98..599) v3 tid 17369) v2 -- ?+0 0x20fcb40 con 0x208a840 -23005> 2013-04-03 10:33:55.651890 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e599 _reply_map 598 from osd.2 10.252.24.30:6801/10373 -23004> 2013-04-03 10:33:55.651911 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_latest to osd.2 10.252.24.30:6801/10373 start 598 -23003> 2013-04-03 10:33:55.651916 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_incremental [598..599] to osd.2 10.252.24.30:6801/10373 -23002> 2013-04-03 10:33:55.651927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 build_incremental [598..599] -23001> 2013-04-03 10:33:55.651957 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(598..599 src has 98..599) v3 tid 17370) v2 -- ?+0 0x20add80 con 0x208a840 -23000> 2013-04-03 10:33:55.651967 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e599 _reply_map 598 from osd.0 10.252.58.31:6802/2682 -22999> 2013-04-03 10:33:55.651985 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_latest to osd.0 10.252.58.31:6802/2682 start 598 -22998> 2013-04-03 10:33:55.651990 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_incremental [598..599] to osd.0 10.252.58.31:6802/2682 -22997> 2013-04-03 10:33:55.652000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 build_incremental [598..599] -22996> 2013-04-03 10:33:55.652028 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(598..599 src has 98..599) v3 tid 17371) v2 -- ?+0 0x20adb40 con 0x208a840 -22995> 2013-04-03 10:33:55.652038 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e599 _reply_map 598 from osd.3 10.252.24.88:6801/17538 -22994> 2013-04-03 10:33:55.652057 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_latest to osd.3 10.252.24.88:6801/17538 start 598 -22993> 2013-04-03 10:33:55.652062 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_incremental [598..599] to osd.3 10.252.24.88:6801/17538 -22992> 2013-04-03 10:33:55.652072 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 build_incremental [598..599] -22991> 2013-04-03 10:33:55.652100 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(598..599 src has 98..599) v3 tid 17373) v2 -- ?+0 0x20fcd80 con 0x208a840 -22990> 2013-04-03 10:33:55.652109 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e599 _reply_map 598 from osd.1 10.252.24.25:6805/3572 -22989> 2013-04-03 10:33:55.652132 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_latest to osd.1 10.252.24.25:6805/3572 start 598 -22988> 2013-04-03 10:33:55.652138 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e599 send_incremental [598..599] to osd.1 10.252.24.25:6805/3572 -22987> 2013-04-03 10:33:55.652148 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 build_incremental [598..599] -22986> 2013-04-03 10:33:55.652178 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(598..599 src has 98..599) v3 tid 17374) v2 -- ?+0 0x20fc240 con 0x208a840 -22985> 2013-04-03 10:33:55.652190 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e599 update_logger -22984> 2013-04-03 10:33:55.652199 7fcdffcdb700 0 log [INF] : osdmap e599: 5 osds: 5 up, 5 in -22983> 2013-04-03 10:33:55.652201 7fcdffcdb700 10 send_log to self -22982> 2013-04-03 10:33:55.652201 7fcdffcdb700 10 log_queue is 2 last_log 13 sent 11 num 2 unsent 2 sending 2 -22981> 2013-04-03 10:33:55.652203 7fcdffcdb700 10 will send 2013-04-03 10:33:54.673122 mon.0 10.252.24.25:6789/0 12 : [INF] pgmap v28960: 240 pgs: 190 active+clean, 38 stale+active+clean, 12 peering; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail -22980> 2013-04-03 10:33:55.652209 7fcdffcdb700 10 will send 2013-04-03 10:33:55.652200 mon.0 10.252.24.25:6789/0 13 : [INF] osdmap e599: 5 osds: 5 up, 5 in -22979> 2013-04-03 10:33:55.652213 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x209c200 -22978> 2013-04-03 10:33:55.652238 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427161..427197) propose_queued 427198 17050 bytes -22977> 2013-04-03 10:33:55.653419 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427197 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4780 -22976> 2013-04-03 10:33:55.653457 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x209c200 con 0x1bac160 -22975> 2013-04-03 10:33:55.653484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos -22974> 2013-04-03 10:33:55.653499 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos version 393968 summary v 393968 -22973> 2013-04-03 10:33:55.653508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -22972> 2013-04-03 10:33:55.653520 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 preprocess_log log(2 entries) v1 from mon.0 -22971> 2013-04-03 10:33:55.653545 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -22970> 2013-04-03 10:33:55.653557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 prepare_log log(2 entries) v1 from mon.0 -22969> 2013-04-03 10:33:55.653566 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 logging 2013-04-03 10:33:54.673122 mon.0 10.252.24.25:6789/0 12 : [INF] pgmap v28960: 240 pgs: 190 active+clean, 38 stale+active+clean, 12 peering; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail -22968> 2013-04-03 10:33:55.653585 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 logging 2013-04-03 10:33:55.652200 mon.0 10.252.24.25:6789/0 13 : [INF] osdmap e599: 5 osds: 5 up, 5 in -22967> 2013-04-03 10:33:55.663279 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112878 ==== paxos(lease_ack lc 427197 fc 427161 pn 0 opn 0) v3 ==== 80+0+0 (1890332793 0 0) 0x20f4280 con 0x208a840 -22966> 2013-04-03 10:33:55.868067 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112879 ==== paxos(accept lc 427197 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (887617845 0 0) 0x20f4780 con 0x208a840 -22965> 2013-04-03 10:33:55.977329 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427198 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4280 -22964> 2013-04-03 10:33:55.977364 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427198 fc 427161 pn 0 opn 0) v3 -- ?+0 0x2041000 -22963> 2013-04-03 10:33:55.977494 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28960 update_from_paxos applying incremental 28961 -22962> 2013-04-03 10:33:55.977623 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 v28961: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail -22961> 2013-04-03 10:33:55.977650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 send_pg_creates to 0 pgs -22960> 2013-04-03 10:33:55.977655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 update_logger -22959> 2013-04-03 10:33:55.977672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 create_pending v 28962 -22958> 2013-04-03 10:33:55.977682 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28961 _updated_stats for osd.1 10.252.24.25:6805/3572 -22957> 2013-04-03 10:33:55.977690 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(45 pgs tid 1) v1 tid 17375) v2 -- ?+0 0x20fc6c0 con 0x208a840 -22956> 2013-04-03 10:33:55.977730 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 check_osd_map applying osdmap e599 to pg_map -22955> 2013-04-03 10:33:55.977752 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 register_new_pgs checking pg pools for osdmap epoch 599, last_pg_scan 341 -22954> 2013-04-03 10:33:55.977756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 -22953> 2013-04-03 10:33:55.977760 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -22952> 2013-04-03 10:33:55.977763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -22951> 2013-04-03 10:33:55.977765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 -22950> 2013-04-03 10:33:55.977768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 -22949> 2013-04-03 10:33:55.977770 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 -22948> 2013-04-03 10:33:55.977773 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 -22947> 2013-04-03 10:33:55.977777 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 -22946> 2013-04-03 10:33:55.977779 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 -22945> 2013-04-03 10:33:55.977792 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs -22944> 2013-04-03 10:33:55.977794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 check_down_pgs -22943> 2013-04-03 10:33:55.977845 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 encode_pending v 28962 -22942> 2013-04-03 10:33:55.977867 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427161..427198) queue_proposal bl 215 bytes; ctx = 0x1b914e0 -22941> 2013-04-03 10:33:55.977875 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427161..427198) propose_queued 427199 215 bytes -22940> 2013-04-03 10:33:56.014023 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427198 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116780 -22939> 2013-04-03 10:33:56.014058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28961 send_pg_creates to 0 pgs -22938> 2013-04-03 10:33:56.046074 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112880 ==== paxos(lease_ack lc 427198 fc 427161 pn 0 opn 0) v3 ==== 80+0+0 (1932892858 0 0) 0x2116780 con 0x208a840 -22937> 2013-04-03 10:33:56.129406 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112881 ==== paxos(accept lc 427198 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3252281338 0 0) 0x2041000 con 0x208a840 -22936> 2013-04-03 10:33:56.135768 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427199 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116780 -22935> 2013-04-03 10:33:56.135787 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427199 fc 427161 pn 0 opn 0) v3 -- ?+0 0x20f4780 -22934> 2013-04-03 10:33:56.135870 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28961 update_from_paxos applying incremental 28962 -22933> 2013-04-03 10:33:56.135880 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 v28962: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail -22932> 2013-04-03 10:33:56.135897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 send_pg_creates to 0 pgs -22931> 2013-04-03 10:33:56.135900 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 update_logger -22930> 2013-04-03 10:33:56.135908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 create_pending v 28963 -22929> 2013-04-03 10:33:56.135916 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 check_osd_map already seen 599 >= 599 -22928> 2013-04-03 10:33:56.135920 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 update_logger -22927> 2013-04-03 10:33:56.135932 7fcdffcdb700 0 log [INF] : pgmap v28962: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail -22926> 2013-04-03 10:33:56.135933 7fcdffcdb700 10 send_log to self -22925> 2013-04-03 10:33:56.135934 7fcdffcdb700 10 log_queue is 3 last_log 14 sent 13 num 3 unsent 1 sending 1 -22924> 2013-04-03 10:33:56.135936 7fcdffcdb700 10 will send 2013-04-03 10:33:55.652200 mon.0 10.252.24.25:6789/0 13 : [INF] osdmap e599: 5 osds: 5 up, 5 in -22923> 2013-04-03 10:33:56.135943 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x209ce00 -22922> 2013-04-03 10:33:56.135987 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos -22921> 2013-04-03 10:33:56.136006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos version 393968 summary v 393968 -22920> 2013-04-03 10:33:56.136025 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -22919> 2013-04-03 10:33:56.136044 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x209ce00 con 0x1bac160 -22918> 2013-04-03 10:33:56.136065 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos -22917> 2013-04-03 10:33:56.136080 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos version 393968 summary v 393968 -22916> 2013-04-03 10:33:56.136089 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -22915> 2013-04-03 10:33:56.136099 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 preprocess_log log(1 entries) v1 from mon.0 -22914> 2013-04-03 10:33:56.136120 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 nothing new -22913> 2013-04-03 10:33:56.162984 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112882 ==== paxos(lease_ack lc 427199 fc 427161 pn 0 opn 0) v3 ==== 80+0+0 (927802953 0 0) 0x20f4780 con 0x208a840 -22912> 2013-04-03 10:33:56.163044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos -22911> 2013-04-03 10:33:56.163063 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393968 update_from_paxos version 393968 summary v 393968 -22910> 2013-04-03 10:33:56.163078 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -22909> 2013-04-03 10:33:56.650245 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393968 encode_pending v393969 -22908> 2013-04-03 10:33:56.650273 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427161..427199) queue_proposal bl 663 bytes; ctx = 0x1b915b0 -22907> 2013-04-03 10:33:56.650281 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427161..427199) propose_queued 427200 663 bytes -22906> 2013-04-03 10:33:56.662101 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427199 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041a00 -22905> 2013-04-03 10:33:56.681136 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112883 ==== paxos(accept lc 427199 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3343906839 0 0) 0x2041a00 con 0x208a840 -22904> 2013-04-03 10:33:56.695947 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427200 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4780 -22903> 2013-04-03 10:33:56.695980 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427200 fc 427161 pn 0 opn 0) v3 -- ?+0 0x2041000 -22902> 2013-04-03 10:33:56.696016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos -22901> 2013-04-03 10:33:56.696040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393968 -22900> 2013-04-03 10:33:56.696054 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos latest full 393798 -22899> 2013-04-03 10:33:56.696073 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393969 update_from_paxos applying incremental log 393969 2013-04-03 10:33:54.673122 mon.0 10.252.24.25:6789/0 12 : [INF] pgmap v28960: 240 pgs: 190 active+clean, 38 stale+active+clean, 12 peering; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail -22898> 2013-04-03 10:33:56.696099 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393969 update_from_paxos applying incremental log 393969 2013-04-03 10:33:55.652200 mon.0 10.252.24.25:6789/0 13 : [INF] osdmap e599: 5 osds: 5 up, 5 in -22897> 2013-04-03 10:33:56.696137 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 check_subs -22896> 2013-04-03 10:33:56.696165 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 create_pending v 393970 -22895> 2013-04-03 10:33:56.696186 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393969 _updated_log for mon.0 10.252.24.25:6789/0 -22894> 2013-04-03 10:33:56.696199 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 13) v1 -- ?+0 0x2224380 con 0x1bac160 -22893> 2013-04-03 10:33:56.696245 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos -22892> 2013-04-03 10:33:56.696260 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 -22891> 2013-04-03 10:33:56.696277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -22890> 2013-04-03 10:33:56.696297 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 13) v1 ==== 0+0+0 (0 0 0) 0x2224380 con 0x1bac160 -22889> 2013-04-03 10:33:56.696305 7fcdffcdb700 10 handle_log_ack log(last 13) v1 -22888> 2013-04-03 10:33:56.696307 7fcdffcdb700 10 logged 2013-04-03 10:33:54.673122 mon.0 10.252.24.25:6789/0 12 : [INF] pgmap v28960: 240 pgs: 190 active+clean, 38 stale+active+clean, 12 peering; 258 MB data, 5671 MB used, 1067 GB / 1079 GB avail -22887> 2013-04-03 10:33:56.696313 7fcdffcdb700 10 logged 2013-04-03 10:33:55.652200 mon.0 10.252.24.25:6789/0 13 : [INF] osdmap e599: 5 osds: 5 up, 5 in -22886> 2013-04-03 10:33:56.733348 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112884 ==== paxos(lease_ack lc 427200 fc 427161 pn 0 opn 0) v3 ==== 80+0+0 (3145307700 0 0) 0x2041000 con 0x208a840 -22885> 2013-04-03 10:33:56.733389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos -22884> 2013-04-03 10:33:56.733405 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 -22883> 2013-04-03 10:33:56.733419 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -22882> 2013-04-03 10:33:57.375695 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 4 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21663 v184) v2 ==== 289+0+0 (2561066627 0 0) 0x1bae340 con 0x200d420 -22881> 2013-04-03 10:33:57.375769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21663 v184) v2 from mds.? 10.252.24.88:6800/17213 -22880> 2013-04-03 10:33:57.375782 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21663 v184) v2 -- ?+0 0x1bae8c0 con 0x200d420 -22879> 2013-04-03 10:33:58.318478 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28962 check_down_pgs -22878> 2013-04-03 10:33:58.318505 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28962 v28962: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail -22877> 2013-04-03 10:33:58.318535 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e184 e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby -22876> 2013-04-03 10:33:58.318560 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in -22875> 2013-04-03 10:33:58.318596 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 -22874> 2013-04-03 10:33:58.318625 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos -22873> 2013-04-03 10:33:58.318642 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 -22872> 2013-04-03 10:33:58.318651 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393969 log -22871> 2013-04-03 10:33:58.318667 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2059 update_from_paxos -22870> 2013-04-03 10:33:58.318684 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2059 auth -22869> 2013-04-03 10:33:58.318703 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2059 AuthMonitor::tick() updated rotating, now calling propose_pending -22868> 2013-04-03 10:33:58.318740 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2059 encode_full auth v 2059 -22867> 2013-04-03 10:33:58.318827 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2059 encode_pending v 2060 -22866> 2013-04-03 10:33:58.318874 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427161..427200) queue_proposal bl 2779 bytes; ctx = 0x1b914c0 -22865> 2013-04-03 10:33:58.318887 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427161..427200) propose_queued 427201 2779 bytes -22864> 2013-04-03 10:33:58.321001 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427200 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116000 -22863> 2013-04-03 10:33:58.357787 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112885 ==== paxos(accept lc 427200 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4280481521 0 0) 0x2116000 con 0x208a840 -22862> 2013-04-03 10:33:58.377334 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427201 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041000 -22861> 2013-04-03 10:33:58.377358 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427201 fc 427161 pn 0 opn 0) v3 -- ?+0 0x2041a00 -22860> 2013-04-03 10:33:58.377413 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22859> 2013-04-03 10:33:58.377448 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos version 2060 keys ver 2059 latest 2059 -22858> 2013-04-03 10:33:58.377461 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos key server version 2059 -22857> 2013-04-03 10:33:58.377493 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos() last_allocated_id=11896 max_global_id=11896 -22856> 2013-04-03 10:33:58.377511 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 create_pending v 2061 -22855> 2013-04-03 10:33:58.377532 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 AuthMonitor::on_active() -22854> 2013-04-03 10:33:58.377575 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos -22853> 2013-04-03 10:33:58.377591 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 -22852> 2013-04-03 10:33:58.377606 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22851> 2013-04-03 10:33:58.417268 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112886 ==== paxos(lease_ack lc 427201 fc 427161 pn 0 opn 0) v3 ==== 80+0+0 (3444563647 0 0) 0x2041a00 con 0x208a840 -22850> 2013-04-03 10:33:58.417310 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos -22849> 2013-04-03 10:33:58.417333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 -22848> 2013-04-03 10:33:58.417347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22847> 2013-04-03 10:33:58.754728 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112887 ==== forward(mdsbeacon(11234/hotcpc9039 up:replay seq 5 v184) v2) to leader v1 ==== 560+0+0 (3701071085 0 0) 0x20e3dc0 con 0x208a840 -22846> 2013-04-03 10:33:58.754745 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -22845> 2013-04-03 10:33:58.754754 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e3b00 from 10.252.24.88:6789/0 -22844> 2013-04-03 10:33:58.754787 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 5 v184) v2 from mds.0 10.252.24.25:6802/3144 -22843> 2013-04-03 10:33:58.754797 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:replay seq 5 v184) v2 tid 17376) v2 -- ?+0 0x20fc480 con 0x208a840 -22842> 2013-04-03 10:33:59.399599 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112888 ==== forward(pg_stats(0 pgs tid 56 v 599) v1) to leader v1 ==== 368+0+0 (2993256779 0 0) 0x1bae340 con 0x208a840 -22841> 2013-04-03 10:33:59.399627 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 -22840> 2013-04-03 10:33:59.399637 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc480 from 10.252.24.88:6789/0 -22839> 2013-04-03 10:33:59.399686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 preprocess_query pg_stats(0 pgs tid 56 v 599) v1 from osd.4 10.252.24.32:6800/13473 -22838> 2013-04-03 10:33:59.399696 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 prepare_update pg_stats(0 pgs tid 56 v 599) v1 from osd.4 10.252.24.32:6800/13473 -22837> 2013-04-03 10:33:59.399699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 prepare_pg_stats pg_stats(0 pgs tid 56 v 599) v1 from osd.4 -22836> 2013-04-03 10:33:59.399706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28962 got osd.4 osd_stat(1052 MB used, 56605 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1052 MB used, 56605 MB avail, 59728 MB total, peers []/[])) -22835> 2013-04-03 10:33:59.449885 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28962 encode_pending v 28963 -22834> 2013-04-03 10:33:59.449918 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427161..427201) queue_proposal bl 265 bytes; ctx = 0x1b91500 -22833> 2013-04-03 10:33:59.449929 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427161..427201) propose_queued 427202 265 bytes -22832> 2013-04-03 10:33:59.451117 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427201 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116280 -22831> 2013-04-03 10:33:59.492186 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112889 ==== paxos(accept lc 427201 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (40416252 0 0) 0x2116280 con 0x208a840 -22830> 2013-04-03 10:33:59.492476 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427202 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041a00 -22829> 2013-04-03 10:33:59.492512 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427202 fc 427161 pn 0 opn 0) v3 -- ?+0 0x2116000 -22828> 2013-04-03 10:33:59.492578 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28962 update_from_paxos applying incremental 28963 -22827> 2013-04-03 10:33:59.492587 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 v28963: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail -22826> 2013-04-03 10:33:59.492604 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 send_pg_creates to 0 pgs -22825> 2013-04-03 10:33:59.492608 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 update_logger -22824> 2013-04-03 10:33:59.492617 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 create_pending v 28964 -22823> 2013-04-03 10:33:59.492626 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28963 _updated_stats for osd.4 10.252.24.32:6800/13473 -22822> 2013-04-03 10:33:59.492632 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(0 pgs tid 56) v1 tid 17377) v2 -- ?+0 0x20fc6c0 con 0x208a840 -22821> 2013-04-03 10:33:59.492641 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 check_osd_map already seen 599 >= 599 -22820> 2013-04-03 10:33:59.492644 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 update_logger -22819> 2013-04-03 10:33:59.493016 7fcdffcdb700 0 log [INF] : pgmap v28963: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail -22818> 2013-04-03 10:33:59.493020 7fcdffcdb700 10 send_log to self -22817> 2013-04-03 10:33:59.493021 7fcdffcdb700 10 log_queue is 2 last_log 15 sent 13 num 2 unsent 2 sending 2 -22816> 2013-04-03 10:33:59.493023 7fcdffcdb700 10 will send 2013-04-03 10:33:56.135933 mon.0 10.252.24.25:6789/0 14 : [INF] pgmap v28962: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail -22815> 2013-04-03 10:33:59.493031 7fcdffcdb700 10 will send 2013-04-03 10:33:59.493020 mon.0 10.252.24.25:6789/0 15 : [INF] pgmap v28963: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail -22814> 2013-04-03 10:33:59.493036 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x209c200 -22813> 2013-04-03 10:33:59.493113 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427161..427202) queue_proposal bl 987 bytes; ctx = 0x1b91760 -22812> 2013-04-03 10:33:59.493123 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427161..427202) propose_queued 427203 987 bytes -22811> 2013-04-03 10:33:59.493327 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427202 fc 0 pn 17600 opn 0) v3 -- ?+0 0x1f91c80 -22810> 2013-04-03 10:33:59.493361 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x209c200 con 0x1bac160 -22809> 2013-04-03 10:33:59.493400 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos -22808> 2013-04-03 10:33:59.493415 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 -22807> 2013-04-03 10:33:59.493424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -22806> 2013-04-03 10:33:59.493436 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 preprocess_log log(2 entries) v1 from mon.0 -22805> 2013-04-03 10:33:59.493463 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -22804> 2013-04-03 10:33:59.493475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 prepare_log log(2 entries) v1 from mon.0 -22803> 2013-04-03 10:33:59.493486 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 logging 2013-04-03 10:33:56.135933 mon.0 10.252.24.25:6789/0 14 : [INF] pgmap v28962: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail -22802> 2013-04-03 10:33:59.493507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 logging 2013-04-03 10:33:59.493020 mon.0 10.252.24.25:6789/0 15 : [INF] pgmap v28963: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail -22801> 2013-04-03 10:33:59.504544 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112890 ==== paxos(lease_ack lc 427202 fc 427161 pn 0 opn 0) v3 ==== 80+0+0 (2717789241 0 0) 0x1f91c80 con 0x208a840 -22800> 2013-04-03 10:33:59.518168 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112891 ==== paxos(accept lc 427202 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1739394077 0 0) 0x2116000 con 0x208a840 -22799> 2013-04-03 10:33:59.518501 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427203 fc 0 pn 17600 opn 0) v3 -- ?+0 0x1f91c80 -22798> 2013-04-03 10:33:59.518527 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427203 fc 427161 pn 0 opn 0) v3 -- ?+0 0x2116280 -22797> 2013-04-03 10:33:59.518593 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos -22796> 2013-04-03 10:33:59.518611 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 -22795> 2013-04-03 10:33:59.518627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22794> 2013-04-03 10:33:59.544327 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112892 ==== paxos(lease_ack lc 427203 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (2267122381 0 0) 0x2116280 con 0x208a840 -22793> 2013-04-03 10:33:59.544361 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos -22792> 2013-04-03 10:33:59.544377 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393969 update_from_paxos version 393969 summary v 393969 -22791> 2013-04-03 10:33:59.544391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22790> 2013-04-03 10:33:59.575403 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 5 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140293 v184) v2 ==== 289+0+0 (1064873855 0 0) 0x1baeb80 con 0x2118420 -22789> 2013-04-03 10:33:59.575445 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140293 v184) v2 from mds.? 10.252.58.31:6800/10908 -22788> 2013-04-03 10:33:59.575456 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140293 v184) v2 -- ?+0 0x1bae340 con 0x2118420 -22787> 2013-04-03 10:33:59.785904 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112893 ==== forward(pg_stats(51 pgs tid 402 v 599) v1) to leader v1 ==== 19407+0+0 (3858393180 0 0) 0x20e3840 con 0x208a840 -22786> 2013-04-03 10:33:59.785930 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -22785> 2013-04-03 10:33:59.785939 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc6c0 from 10.252.24.88:6789/0 -22784> 2013-04-03 10:33:59.785988 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 preprocess_query pg_stats(51 pgs tid 402 v 599) v1 from osd.0 10.252.58.31:6802/2682 -22783> 2013-04-03 10:33:59.785996 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 prepare_update pg_stats(51 pgs tid 402 v 599) v1 from osd.0 10.252.58.31:6802/2682 -22782> 2013-04-03 10:33:59.785999 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 prepare_pg_stats pg_stats(51 pgs tid 402 v 599) v1 from osd.0 -22781> 2013-04-03 10:33:59.786005 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 got osd.0 osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -22780> 2013-04-03 10:33:59.827570 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112894 ==== forward(pg_stats(71 pgs tid 521 v 599) v1) to leader v1 ==== 26863+0+0 (809306997 0 0) 0x20e3580 con 0x208a840 -22779> 2013-04-03 10:33:59.827582 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 -22778> 2013-04-03 10:33:59.827589 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 -22777> 2013-04-03 10:33:59.827615 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 preprocess_query pg_stats(71 pgs tid 521 v 599) v1 from osd.2 10.252.24.30:6801/10373 -22776> 2013-04-03 10:33:59.827622 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 prepare_update pg_stats(71 pgs tid 521 v 599) v1 from osd.2 10.252.24.30:6801/10373 -22775> 2013-04-03 10:33:59.827625 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 prepare_pg_stats pg_stats(71 pgs tid 521 v 599) v1 from osd.2 -22774> 2013-04-03 10:33:59.827629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28963 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -22773> 2013-04-03 10:34:00.492382 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393969 encode_pending v393970 -22772> 2013-04-03 10:34:00.492420 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427203) queue_proposal bl 712 bytes; ctx = 0x1b914a0 -22771> 2013-04-03 10:34:00.492428 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427203) propose_queued 427204 712 bytes -22770> 2013-04-03 10:34:00.493649 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427203 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116500 -22769> 2013-04-03 10:34:00.518326 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28963 encode_pending v 28964 -22768> 2013-04-03 10:34:00.518660 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427192..427203) queue_proposal bl 45849 bytes; ctx = 0x1b91770 -22767> 2013-04-03 10:34:00.518668 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427192..427203) propose_new_value not active; proposal queued -22766> 2013-04-03 10:34:00.619783 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112895 ==== paxos(accept lc 427203 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (973306189 0 0) 0x2116500 con 0x208a840 -22765> 2013-04-03 10:34:00.620105 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427204 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116280 -22764> 2013-04-03 10:34:00.620138 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427204 fc 427192 pn 0 opn 0) v3 -- ?+0 0x2116000 -22763> 2013-04-03 10:34:00.620184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos -22762> 2013-04-03 10:34:00.620213 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos version 393970 summary v 393969 -22761> 2013-04-03 10:34:00.620227 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos latest full 393798 -22760> 2013-04-03 10:34:00.620245 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393970 update_from_paxos applying incremental log 393970 2013-04-03 10:33:56.135933 mon.0 10.252.24.25:6789/0 14 : [INF] pgmap v28962: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail -22759> 2013-04-03 10:34:00.620270 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393970 update_from_paxos applying incremental log 393970 2013-04-03 10:33:59.493020 mon.0 10.252.24.25:6789/0 15 : [INF] pgmap v28963: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail -22758> 2013-04-03 10:34:00.620309 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 check_subs -22757> 2013-04-03 10:34:00.620337 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 create_pending v 393971 -22756> 2013-04-03 10:34:00.620358 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393970 _updated_log for mon.0 10.252.24.25:6789/0 -22755> 2013-04-03 10:34:00.620371 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 15) v1 -- ?+0 0x22241c0 con 0x1bac160 -22754> 2013-04-03 10:34:00.620396 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427204) propose_queued 427205 45849 bytes -22753> 2013-04-03 10:34:00.622086 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427204 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041780 -22752> 2013-04-03 10:34:00.622106 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 15) v1 ==== 0+0+0 (0 0 0) 0x22241c0 con 0x1bac160 -22751> 2013-04-03 10:34:00.622115 7fcdffcdb700 10 handle_log_ack log(last 15) v1 -22750> 2013-04-03 10:34:00.622116 7fcdffcdb700 10 logged 2013-04-03 10:33:56.135933 mon.0 10.252.24.25:6789/0 14 : [INF] pgmap v28962: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail -22749> 2013-04-03 10:34:00.622123 7fcdffcdb700 10 logged 2013-04-03 10:33:59.493020 mon.0 10.252.24.25:6789/0 15 : [INF] pgmap v28963: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5669 MB used, 1067 GB / 1079 GB avail -22748> 2013-04-03 10:34:00.732427 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112896 ==== paxos(lease_ack lc 427204 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (844256296 0 0) 0x2041780 con 0x208a840 -22747> 2013-04-03 10:34:00.734238 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112897 ==== forward(pg_stats(45 pgs tid 2 v 599) v1) to leader v1 ==== 17169+0+0 (351155128 0 0) 0x20e32c0 con 0x208a840 -22746> 2013-04-03 10:34:00.734251 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -22745> 2013-04-03 10:34:00.734259 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fcd80 from 10.252.24.88:6789/0 -22744> 2013-04-03 10:34:00.920413 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112898 ==== paxos(accept lc 427204 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2119608528 0 0) 0x2116000 con 0x208a840 -22743> 2013-04-03 10:34:00.921919 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427205 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041780 -22742> 2013-04-03 10:34:00.921937 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427205 fc 427192 pn 0 opn 0) v3 -- ?+0 0x2116500 -22741> 2013-04-03 10:34:00.922046 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28963 update_from_paxos applying incremental 28964 -22740> 2013-04-03 10:34:00.922349 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 v28964: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5672 MB used, 1067 GB / 1079 GB avail; 3608B/s wr, 0op/s -22739> 2013-04-03 10:34:00.922384 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 send_pg_creates to 0 pgs -22738> 2013-04-03 10:34:00.922389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 update_logger -22737> 2013-04-03 10:34:00.922415 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 create_pending v 28965 -22736> 2013-04-03 10:34:00.922427 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28964 _updated_stats for osd.0 10.252.58.31:6802/2682 -22735> 2013-04-03 10:34:00.922434 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(51 pgs tid 402) v1 tid 17378) v2 -- ?+0 0x20ad900 con 0x208a840 -22734> 2013-04-03 10:34:00.922453 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28964 _updated_stats for osd.2 10.252.24.30:6801/10373 -22733> 2013-04-03 10:34:00.922457 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(71 pgs tid 521) v1 tid 17379) v2 -- ?+0 0x20fc6c0 con 0x208a840 -22732> 2013-04-03 10:34:00.922523 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 preprocess_query pg_stats(45 pgs tid 2 v 599) v1 from osd.1 10.252.24.25:6805/3572 -22731> 2013-04-03 10:34:00.922531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 prepare_update pg_stats(45 pgs tid 2 v 599) v1 from osd.1 10.252.24.25:6805/3572 -22730> 2013-04-03 10:34:00.922535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 prepare_pg_stats pg_stats(45 pgs tid 2 v 599) v1 from osd.1 -22729> 2013-04-03 10:34:00.922539 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 got osd.1 osd_stat(1113 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1110 MB used, 148 GB avail, 149 GB total, peers []/[])) -22728> 2013-04-03 10:34:00.922595 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 check_osd_map already seen 599 >= 599 -22727> 2013-04-03 10:34:00.922600 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 update_logger -22726> 2013-04-03 10:34:00.922613 7fcdffcdb700 0 log [INF] : pgmap v28964: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5672 MB used, 1067 GB / 1079 GB avail; 3608B/s wr, 0op/s -22725> 2013-04-03 10:34:00.922614 7fcdffcdb700 10 send_log to self -22724> 2013-04-03 10:34:00.922615 7fcdffcdb700 10 log_queue is 1 last_log 16 sent 15 num 1 unsent 1 sending 1 -22723> 2013-04-03 10:34:00.922617 7fcdffcdb700 10 will send 2013-04-03 10:34:00.922614 mon.0 10.252.24.25:6789/0 16 : [INF] pgmap v28964: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5672 MB used, 1067 GB / 1079 GB avail; 3608B/s wr, 0op/s -22722> 2013-04-03 10:34:00.922625 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2169200 -22721> 2013-04-03 10:34:00.922669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos -22720> 2013-04-03 10:34:00.922686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos version 393970 summary v 393970 -22719> 2013-04-03 10:34:00.922703 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22718> 2013-04-03 10:34:00.922724 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2169200 con 0x1bac160 -22717> 2013-04-03 10:34:00.922744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos -22716> 2013-04-03 10:34:00.922759 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos version 393970 summary v 393970 -22715> 2013-04-03 10:34:00.922767 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -22714> 2013-04-03 10:34:00.922779 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 preprocess_log log(1 entries) v1 from mon.0 -22713> 2013-04-03 10:34:00.922797 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -22712> 2013-04-03 10:34:00.922809 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 prepare_log log(1 entries) v1 from mon.0 -22711> 2013-04-03 10:34:00.922820 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 logging 2013-04-03 10:34:00.922614 mon.0 10.252.24.25:6789/0 16 : [INF] pgmap v28964: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5672 MB used, 1067 GB / 1079 GB avail; 3608B/s wr, 0op/s -22710> 2013-04-03 10:34:00.954350 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112899 ==== paxos(lease_ack lc 427205 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (2971960048 0 0) 0x2116500 con 0x208a840 -22709> 2013-04-03 10:34:00.954391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos -22708> 2013-04-03 10:34:00.954409 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393970 update_from_paxos version 393970 summary v 393970 -22707> 2013-04-03 10:34:00.954423 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22706> 2013-04-03 10:34:00.957775 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112900 ==== forward(pg_stats(73 pgs tid 30430 v 599) v1) to leader v1 ==== 27613+0+0 (4023306360 0 0) 0x20e3000 con 0x208a840 -22705> 2013-04-03 10:34:00.957796 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -22704> 2013-04-03 10:34:00.957805 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc6c0 from 10.252.24.88:6789/0 -22703> 2013-04-03 10:34:00.957852 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 preprocess_query pg_stats(73 pgs tid 30430 v 599) v1 from osd.3 10.252.24.88:6801/17538 -22702> 2013-04-03 10:34:00.957860 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 prepare_update pg_stats(73 pgs tid 30430 v 599) v1 from osd.3 10.252.24.88:6801/17538 -22701> 2013-04-03 10:34:00.957863 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 prepare_pg_stats pg_stats(73 pgs tid 30430 v 599) v1 from osd.3 -22700> 2013-04-03 10:34:00.957868 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28964 got osd.3 osd_stat(1190 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1187 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -22699> 2013-04-03 10:34:01.375773 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 5 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21664 v184) v2 ==== 289+0+0 (694376895 0 0) 0x1bae340 con 0x200d420 -22698> 2013-04-03 10:34:01.375848 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21664 v184) v2 from mds.? 10.252.24.88:6800/17213 -22697> 2013-04-03 10:34:01.375863 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21664 v184) v2 -- ?+0 0x20e4080 con 0x200d420 -22696> 2013-04-03 10:34:01.920604 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28964 encode_pending v 28965 -22695> 2013-04-03 10:34:01.920923 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427205) queue_proposal bl 44361 bytes; ctx = 0x1b91740 -22694> 2013-04-03 10:34:01.920934 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427205) propose_queued 427206 44361 bytes -22693> 2013-04-03 10:34:01.922876 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427205 fc 0 pn 17600 opn 0) v3 -- ?+0 0x219ca00 -22692> 2013-04-03 10:34:01.922961 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393970 encode_pending v393971 -22691> 2013-04-03 10:34:01.922989 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427192..427205) queue_proposal bl 455 bytes; ctx = 0x1b915a0 -22690> 2013-04-03 10:34:01.922995 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427192..427205) propose_new_value not active; proposal queued -22689> 2013-04-03 10:34:01.954327 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112901 ==== paxos(accept lc 427205 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1511611626 0 0) 0x219ca00 con 0x208a840 -22688> 2013-04-03 10:34:01.955731 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427206 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116500 -22687> 2013-04-03 10:34:01.955751 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427206 fc 427192 pn 0 opn 0) v3 -- ?+0 0x2116000 -22686> 2013-04-03 10:34:01.955860 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28964 update_from_paxos applying incremental 28965 -22685> 2013-04-03 10:34:01.956150 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 v28965: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 6631B/s wr, 0op/s -22684> 2013-04-03 10:34:01.956184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 send_pg_creates to 0 pgs -22683> 2013-04-03 10:34:01.956188 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 update_logger -22682> 2013-04-03 10:34:01.956211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 create_pending v 28966 -22681> 2013-04-03 10:34:01.956222 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28965 _updated_stats for osd.1 10.252.24.25:6805/3572 -22680> 2013-04-03 10:34:01.956230 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(45 pgs tid 2) v1 tid 17380) v2 -- ?+0 0x20fc240 con 0x208a840 -22679> 2013-04-03 10:34:01.956262 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28965 _updated_stats for osd.3 10.252.24.88:6801/17538 -22678> 2013-04-03 10:34:01.956268 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(73 pgs tid 30430) v1 tid 17381) v2 -- ?+0 0x20fcd80 con 0x208a840 -22677> 2013-04-03 10:34:01.956297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 check_osd_map already seen 599 >= 599 -22676> 2013-04-03 10:34:01.956301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 update_logger -22675> 2013-04-03 10:34:01.956312 7fcdffcdb700 0 log [INF] : pgmap v28965: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 6631B/s wr, 0op/s -22674> 2013-04-03 10:34:01.956313 7fcdffcdb700 10 send_log to self -22673> 2013-04-03 10:34:01.956314 7fcdffcdb700 10 log_queue is 2 last_log 17 sent 16 num 2 unsent 1 sending 1 -22672> 2013-04-03 10:34:01.956316 7fcdffcdb700 10 will send 2013-04-03 10:34:00.922614 mon.0 10.252.24.25:6789/0 16 : [INF] pgmap v28964: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5672 MB used, 1067 GB / 1079 GB avail; 3608B/s wr, 0op/s -22671> 2013-04-03 10:34:01.956325 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2169000 -22670> 2013-04-03 10:34:01.956352 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427206) propose_queued 427207 455 bytes -22669> 2013-04-03 10:34:01.956556 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427206 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041c80 -22668> 2013-04-03 10:34:01.956619 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2169000 con 0x1bac160 -22667> 2013-04-03 10:34:02.097591 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112902 ==== paxos(lease_ack lc 427206 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (4103887402 0 0) 0x2041c80 con 0x208a840 -22666> 2013-04-03 10:34:02.156304 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112903 ==== paxos(accept lc 427206 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (758337798 0 0) 0x2041780 con 0x208a840 -22665> 2013-04-03 10:34:02.157389 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427207 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041c80 -22664> 2013-04-03 10:34:02.157418 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427207 fc 427192 pn 0 opn 0) v3 -- ?+0 0x219ca00 -22663> 2013-04-03 10:34:02.157463 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos -22662> 2013-04-03 10:34:02.157498 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393970 -22661> 2013-04-03 10:34:02.157512 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos latest full 393798 -22660> 2013-04-03 10:34:02.157529 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393971 update_from_paxos applying incremental log 393971 2013-04-03 10:34:00.922614 mon.0 10.252.24.25:6789/0 16 : [INF] pgmap v28964: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5672 MB used, 1067 GB / 1079 GB avail; 3608B/s wr, 0op/s -22659> 2013-04-03 10:34:02.157571 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 check_subs -22658> 2013-04-03 10:34:02.157601 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 create_pending v 393972 -22657> 2013-04-03 10:34:02.157622 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393971 _updated_log for mon.0 10.252.24.25:6789/0 -22656> 2013-04-03 10:34:02.157634 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 16) v1 -- ?+0 0x2294c40 con 0x1bac160 -22655> 2013-04-03 10:34:02.157654 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos -22654> 2013-04-03 10:34:02.157668 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393971 -22653> 2013-04-03 10:34:02.157677 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -22652> 2013-04-03 10:34:02.157689 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 preprocess_log log(1 entries) v1 from mon.0 -22651> 2013-04-03 10:34:02.157707 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 nothing new -22650> 2013-04-03 10:34:02.157755 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos -22649> 2013-04-03 10:34:02.157770 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393971 -22648> 2013-04-03 10:34:02.157787 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22647> 2013-04-03 10:34:02.157807 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 16) v1 ==== 0+0+0 (0 0 0) 0x2294c40 con 0x1bac160 -22646> 2013-04-03 10:34:02.157815 7fcdffcdb700 10 handle_log_ack log(last 16) v1 -22645> 2013-04-03 10:34:02.157817 7fcdffcdb700 10 logged 2013-04-03 10:34:00.922614 mon.0 10.252.24.25:6789/0 16 : [INF] pgmap v28964: 240 pgs: 190 active+clean, 50 peering; 258 MB data, 5672 MB used, 1067 GB / 1079 GB avail; 3608B/s wr, 0op/s -22644> 2013-04-03 10:34:02.232349 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112904 ==== paxos(lease_ack lc 427207 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (2588990620 0 0) 0x219ca00 con 0x208a840 -22643> 2013-04-03 10:34:02.232389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos -22642> 2013-04-03 10:34:02.232405 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393971 -22641> 2013-04-03 10:34:02.232419 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22640> 2013-04-03 10:34:02.754705 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112905 ==== forward(mdsbeacon(11234/hotcpc9039 up:replay seq 6 v184) v2) to leader v1 ==== 560+0+0 (3203048931 0 0) 0x20e4b80 con 0x208a840 -22639> 2013-04-03 10:34:02.754726 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -22638> 2013-04-03 10:34:02.754736 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e48c0 from 10.252.24.88:6789/0 -22637> 2013-04-03 10:34:02.754785 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 6 v184) v2 from mds.0 10.252.24.25:6802/3144 -22636> 2013-04-03 10:34:02.754797 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:replay seq 6 v184) v2 tid 17382) v2 -- ?+0 0x20fc6c0 con 0x208a840 -22635> 2013-04-03 10:34:03.321182 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28965 check_down_pgs -22634> 2013-04-03 10:34:03.321214 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28965 v28965: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 6631B/s wr, 0op/s -22633> 2013-04-03 10:34:03.321239 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e184 e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby -22632> 2013-04-03 10:34:03.321260 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in -22631> 2013-04-03 10:34:03.321298 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 -22630> 2013-04-03 10:34:03.321335 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos -22629> 2013-04-03 10:34:03.321353 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393971 -22628> 2013-04-03 10:34:03.321362 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393971 log -22627> 2013-04-03 10:34:03.321378 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22626> 2013-04-03 10:34:03.321395 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -22625> 2013-04-03 10:34:03.575557 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 6 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140294 v184) v2 ==== 289+0+0 (1523281656 0 0) 0x1bae600 con 0x2118420 -22624> 2013-04-03 10:34:03.575627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140294 v184) v2 from mds.? 10.252.58.31:6800/10908 -22623> 2013-04-03 10:34:03.575639 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140294 v184) v2 -- ?+0 0x20e4b80 con 0x2118420 -22622> 2013-04-03 10:34:04.823340 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112906 ==== forward(pg_stats(1 pgs tid 522 v 599) v1) to leader v1 ==== 753+0+0 (3121813783 0 0) 0x1bae340 con 0x208a840 -22621> 2013-04-03 10:34:04.823363 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 -22620> 2013-04-03 10:34:04.823372 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc6c0 from 10.252.24.88:6789/0 -22619> 2013-04-03 10:34:04.823421 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 preprocess_query pg_stats(1 pgs tid 522 v 599) v1 from osd.2 10.252.24.30:6801/10373 -22618> 2013-04-03 10:34:04.823429 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 prepare_update pg_stats(1 pgs tid 522 v 599) v1 from osd.2 10.252.24.30:6801/10373 -22617> 2013-04-03 10:34:04.823433 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 prepare_pg_stats pg_stats(1 pgs tid 522 v 599) v1 from osd.2 -22616> 2013-04-03 10:34:04.823438 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28965 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[])) -22615> 2013-04-03 10:34:04.873625 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28965 encode_pending v 28966 -22614> 2013-04-03 10:34:04.873655 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427207) queue_proposal bl 650 bytes; ctx = 0x1b91570 -22613> 2013-04-03 10:34:04.873663 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427207) propose_queued 427208 650 bytes -22612> 2013-04-03 10:34:04.875572 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427207 fc 0 pn 17600 opn 0) v3 -- ?+0 0x219c780 -22611> 2013-04-03 10:34:04.901048 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112907 ==== paxos(accept lc 427207 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2550075576 0 0) 0x219c780 con 0x208a840 -22610> 2013-04-03 10:34:04.901988 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427208 fc 0 pn 17600 opn 0) v3 -- ?+0 0x219ca00 -22609> 2013-04-03 10:34:04.902017 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427208 fc 427192 pn 0 opn 0) v3 -- ?+0 0x2041780 -22608> 2013-04-03 10:34:04.902090 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28965 update_from_paxos applying incremental 28966 -22607> 2013-04-03 10:34:04.902106 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 v28966: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 940B/s wr, 0op/s -22606> 2013-04-03 10:34:04.902124 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 send_pg_creates to 0 pgs -22605> 2013-04-03 10:34:04.902128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 update_logger -22604> 2013-04-03 10:34:04.902138 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 create_pending v 28967 -22603> 2013-04-03 10:34:04.902146 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28966 _updated_stats for osd.2 10.252.24.30:6801/10373 -22602> 2013-04-03 10:34:04.902153 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 522) v1 tid 17383) v2 -- ?+0 0x20fcd80 con 0x208a840 -22601> 2013-04-03 10:34:04.902162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 check_osd_map already seen 599 >= 599 -22600> 2013-04-03 10:34:04.902164 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 update_logger -22599> 2013-04-03 10:34:04.902176 7fcdffcdb700 0 log [INF] : pgmap v28966: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 940B/s wr, 0op/s -22598> 2013-04-03 10:34:04.902178 7fcdffcdb700 10 send_log to self -22597> 2013-04-03 10:34:04.902178 7fcdffcdb700 10 log_queue is 2 last_log 18 sent 16 num 2 unsent 2 sending 2 -22596> 2013-04-03 10:34:04.902180 7fcdffcdb700 10 will send 2013-04-03 10:34:01.956313 mon.0 10.252.24.25:6789/0 17 : [INF] pgmap v28965: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 6631B/s wr, 0op/s -22595> 2013-04-03 10:34:04.902188 7fcdffcdb700 10 will send 2013-04-03 10:34:04.902177 mon.0 10.252.24.25:6789/0 18 : [INF] pgmap v28966: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 940B/s wr, 0op/s -22594> 2013-04-03 10:34:04.902192 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2169000 -22593> 2013-04-03 10:34:04.902241 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos -22592> 2013-04-03 10:34:04.902258 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393971 -22591> 2013-04-03 10:34:04.902276 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22590> 2013-04-03 10:34:04.902297 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2169000 con 0x1bac160 -22589> 2013-04-03 10:34:04.902319 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos -22588> 2013-04-03 10:34:04.902334 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393971 -22587> 2013-04-03 10:34:04.902343 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -22586> 2013-04-03 10:34:04.902353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 preprocess_log log(2 entries) v1 from mon.0 -22585> 2013-04-03 10:34:04.902380 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -22584> 2013-04-03 10:34:04.902391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 prepare_log log(2 entries) v1 from mon.0 -22583> 2013-04-03 10:34:04.902401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 logging 2013-04-03 10:34:01.956313 mon.0 10.252.24.25:6789/0 17 : [INF] pgmap v28965: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 6631B/s wr, 0op/s -22582> 2013-04-03 10:34:04.902424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 logging 2013-04-03 10:34:04.902177 mon.0 10.252.24.25:6789/0 18 : [INF] pgmap v28966: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 940B/s wr, 0op/s -22581> 2013-04-03 10:34:04.923265 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112908 ==== paxos(lease_ack lc 427208 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (74500497 0 0) 0x2041280 con 0x208a840 -22580> 2013-04-03 10:34:04.923298 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos -22579> 2013-04-03 10:34:04.923314 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393971 update_from_paxos version 393971 summary v 393971 -22578> 2013-04-03 10:34:04.923329 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22577> 2013-04-03 10:34:05.375900 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 6 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21665 v184) v2 ==== 289+0+0 (168683202 0 0) 0x1baeb80 con 0x200d420 -22576> 2013-04-03 10:34:05.375972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21665 v184) v2 from mds.? 10.252.24.88:6800/17213 -22575> 2013-04-03 10:34:05.375985 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21665 v184) v2 -- ?+0 0x1bae340 con 0x200d420 -22574> 2013-04-03 10:34:05.901241 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393971 encode_pending v393972 -22573> 2013-04-03 10:34:05.901268 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427208) queue_proposal bl 725 bytes; ctx = 0x1b917a0 -22572> 2013-04-03 10:34:05.901275 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427208) propose_queued 427209 725 bytes -22571> 2013-04-03 10:34:05.902482 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427208 fc 0 pn 17600 opn 0) v3 -- ?+0 0x219c500 -22570> 2013-04-03 10:34:06.071558 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112909 ==== paxos(accept lc 427208 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (82708503 0 0) 0x219c500 con 0x208a840 -22569> 2013-04-03 10:34:06.072307 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427209 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041280 -22568> 2013-04-03 10:34:06.072343 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427209 fc 427192 pn 0 opn 0) v3 -- ?+0 0x219c780 -22567> 2013-04-03 10:34:06.072384 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos -22566> 2013-04-03 10:34:06.072418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393971 -22565> 2013-04-03 10:34:06.072434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos latest full 393798 -22564> 2013-04-03 10:34:06.072454 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393972 update_from_paxos applying incremental log 393972 2013-04-03 10:34:01.956313 mon.0 10.252.24.25:6789/0 17 : [INF] pgmap v28965: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 6631B/s wr, 0op/s -22563> 2013-04-03 10:34:06.072479 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393972 update_from_paxos applying incremental log 393972 2013-04-03 10:34:04.902177 mon.0 10.252.24.25:6789/0 18 : [INF] pgmap v28966: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 940B/s wr, 0op/s -22562> 2013-04-03 10:34:06.072514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 check_subs -22561> 2013-04-03 10:34:06.072543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 create_pending v 393973 -22560> 2013-04-03 10:34:06.072565 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393972 _updated_log for mon.0 10.252.24.25:6789/0 -22559> 2013-04-03 10:34:06.072578 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 18) v1 -- ?+0 0x2294700 con 0x1bac160 -22558> 2013-04-03 10:34:06.072625 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos -22557> 2013-04-03 10:34:06.072640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393972 -22556> 2013-04-03 10:34:06.072657 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22555> 2013-04-03 10:34:06.072677 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 18) v1 ==== 0+0+0 (0 0 0) 0x2294700 con 0x1bac160 -22554> 2013-04-03 10:34:06.072685 7fcdffcdb700 10 handle_log_ack log(last 18) v1 -22553> 2013-04-03 10:34:06.072687 7fcdffcdb700 10 logged 2013-04-03 10:34:01.956313 mon.0 10.252.24.25:6789/0 17 : [INF] pgmap v28965: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 6631B/s wr, 0op/s -22552> 2013-04-03 10:34:06.072693 7fcdffcdb700 10 logged 2013-04-03 10:34:04.902177 mon.0 10.252.24.25:6789/0 18 : [INF] pgmap v28966: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 940B/s wr, 0op/s -22551> 2013-04-03 10:34:06.097570 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112910 ==== paxos(lease_ack lc 427209 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (1719777489 0 0) 0x219c780 con 0x208a840 -22550> 2013-04-03 10:34:06.097612 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos -22549> 2013-04-03 10:34:06.097628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393972 -22548> 2013-04-03 10:34:06.097642 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22547> 2013-04-03 10:34:06.754956 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112911 ==== forward(mdsbeacon(11234/hotcpc9039 up:replay seq 7 v184) v2) to leader v1 ==== 560+0+0 (1029199613 0 0) 0x20e4600 con 0x208a840 -22546> 2013-04-03 10:34:06.754979 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -22545> 2013-04-03 10:34:06.754990 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4340 from 10.252.24.88:6789/0 -22544> 2013-04-03 10:34:06.755039 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 7 v184) v2 from mds.0 10.252.24.25:6802/3144 -22543> 2013-04-03 10:34:06.755050 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:replay seq 7 v184) v2 tid 17384) v2 -- ?+0 0x20ad240 con 0x208a840 -22542> 2013-04-03 10:34:07.575696 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 7 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140295 v184) v2 ==== 289+0+0 (2040910213 0 0) 0x2145580 con 0x2118420 -22541> 2013-04-03 10:34:07.575767 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140295 v184) v2 from mds.? 10.252.58.31:6800/10908 -22540> 2013-04-03 10:34:07.575780 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140295 v184) v2 -- ?+0 0x20e4600 con 0x2118420 -22539> 2013-04-03 10:34:08.321534 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28966 check_down_pgs -22538> 2013-04-03 10:34:08.321572 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28966 v28966: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 940B/s wr, 0op/s -22537> 2013-04-03 10:34:08.321598 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e184 e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby -22536> 2013-04-03 10:34:08.321619 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in -22535> 2013-04-03 10:34:08.321653 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 -22534> 2013-04-03 10:34:08.321686 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos -22533> 2013-04-03 10:34:08.321704 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393972 -22532> 2013-04-03 10:34:08.321713 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393972 log -22531> 2013-04-03 10:34:08.321729 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22530> 2013-04-03 10:34:08.321747 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -22529> 2013-04-03 10:34:09.072417 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427209 fc 427192 pn 0 opn 0) v3 -- ?+0 0x219c280 -22528> 2013-04-03 10:34:09.097059 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112912 ==== paxos(lease_ack lc 427209 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (878874297 0 0) 0x219c280 con 0x208a840 -22527> 2013-04-03 10:34:09.097126 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos -22526> 2013-04-03 10:34:09.097145 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393972 -22525> 2013-04-03 10:34:09.097161 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22524> 2013-04-03 10:34:09.394463 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 7 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21666 v184) v2 ==== 289+0+0 (1874732869 0 0) 0x2145dc0 con 0x200d420 -22523> 2013-04-03 10:34:09.394536 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21666 v184) v2 from mds.? 10.252.24.88:6800/17213 -22522> 2013-04-03 10:34:09.394549 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21666 v184) v2 -- ?+0 0x2145580 con 0x200d420 -22521> 2013-04-03 10:34:09.807000 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112913 ==== forward(pg_stats(4 pgs tid 403 v 599) v1) to leader v1 ==== 1876+0+0 (708155981 0 0) 0x1baeb80 con 0x208a840 -22520> 2013-04-03 10:34:09.807027 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -22519> 2013-04-03 10:34:09.807037 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad240 from 10.252.24.88:6789/0 -22518> 2013-04-03 10:34:09.807088 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 preprocess_query pg_stats(4 pgs tid 403 v 599) v1 from osd.0 10.252.58.31:6802/2682 -22517> 2013-04-03 10:34:09.807095 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 prepare_update pg_stats(4 pgs tid 403 v 599) v1 from osd.0 10.252.58.31:6802/2682 -22516> 2013-04-03 10:34:09.807100 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 prepare_pg_stats pg_stats(4 pgs tid 403 v 599) v1 from osd.0 -22515> 2013-04-03 10:34:09.807106 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 got osd.0 osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -22514> 2013-04-03 10:34:09.847060 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112914 ==== forward(pg_stats(3 pgs tid 523 v 599) v1) to leader v1 ==== 1499+0+0 (522693156 0 0) 0x20e4080 con 0x208a840 -22513> 2013-04-03 10:34:09.847091 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 -22512> 2013-04-03 10:34:09.847100 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 -22511> 2013-04-03 10:34:09.847149 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 preprocess_query pg_stats(3 pgs tid 523 v 599) v1 from osd.2 10.252.24.30:6801/10373 -22510> 2013-04-03 10:34:09.847156 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 prepare_update pg_stats(3 pgs tid 523 v 599) v1 from osd.2 10.252.24.30:6801/10373 -22509> 2013-04-03 10:34:09.847160 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 prepare_pg_stats pg_stats(3 pgs tid 523 v 599) v1 from osd.2 -22508> 2013-04-03 10:34:09.847165 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28966 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[])) -22507> 2013-04-03 10:34:09.857252 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28966 encode_pending v 28967 -22506> 2013-04-03 10:34:09.857299 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427209) queue_proposal bl 2954 bytes; ctx = 0x1b91730 -22505> 2013-04-03 10:34:09.857308 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427209) propose_queued 427210 2954 bytes -22504> 2013-04-03 10:34:09.860166 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427209 fc 0 pn 17600 opn 0) v3 -- ?+0 0x219c000 -22503> 2013-04-03 10:34:09.938815 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112915 ==== paxos(accept lc 427209 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3704402198 0 0) 0x219c000 con 0x208a840 -22502> 2013-04-03 10:34:09.939173 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427210 fc 0 pn 17600 opn 0) v3 -- ?+0 0x219c280 -22501> 2013-04-03 10:34:09.939195 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427210 fc 427192 pn 0 opn 0) v3 -- ?+0 0x219c780 -22500> 2013-04-03 10:34:09.939299 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28966 update_from_paxos applying incremental 28967 -22499> 2013-04-03 10:34:09.939338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 v28967: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 516B/s wr, 0op/s -22498> 2013-04-03 10:34:09.939358 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 send_pg_creates to 0 pgs -22497> 2013-04-03 10:34:09.939363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 update_logger -22496> 2013-04-03 10:34:09.939376 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 create_pending v 28968 -22495> 2013-04-03 10:34:09.939388 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28967 _updated_stats for osd.0 10.252.58.31:6802/2682 -22494> 2013-04-03 10:34:09.939396 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(4 pgs tid 403) v1 tid 17385) v2 -- ?+0 0x20fcd80 con 0x208a840 -22493> 2013-04-03 10:34:09.939407 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28967 _updated_stats for osd.2 10.252.24.30:6801/10373 -22492> 2013-04-03 10:34:09.939411 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(3 pgs tid 523) v1 tid 17386) v2 -- ?+0 0x20ad240 con 0x208a840 -22491> 2013-04-03 10:34:09.939417 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 check_osd_map already seen 599 >= 599 -22490> 2013-04-03 10:34:09.939421 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 update_logger -22489> 2013-04-03 10:34:09.939431 7fcdffcdb700 0 log [INF] : pgmap v28967: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 516B/s wr, 0op/s -22488> 2013-04-03 10:34:09.939432 7fcdffcdb700 10 send_log to self -22487> 2013-04-03 10:34:09.939433 7fcdffcdb700 10 log_queue is 1 last_log 19 sent 18 num 1 unsent 1 sending 1 -22486> 2013-04-03 10:34:09.939435 7fcdffcdb700 10 will send 2013-04-03 10:34:09.939432 mon.0 10.252.24.25:6789/0 19 : [INF] pgmap v28967: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 516B/s wr, 0op/s -22485> 2013-04-03 10:34:09.939442 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x25fce00 -22484> 2013-04-03 10:34:09.939488 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos -22483> 2013-04-03 10:34:09.939507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393972 -22482> 2013-04-03 10:34:09.939523 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22481> 2013-04-03 10:34:09.939541 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x25fce00 con 0x1bac160 -22480> 2013-04-03 10:34:09.939563 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos -22479> 2013-04-03 10:34:09.939578 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393972 -22478> 2013-04-03 10:34:09.939585 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -22477> 2013-04-03 10:34:09.939597 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 preprocess_log log(1 entries) v1 from mon.0 -22476> 2013-04-03 10:34:09.939616 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -22475> 2013-04-03 10:34:09.939627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 prepare_log log(1 entries) v1 from mon.0 -22474> 2013-04-03 10:34:09.939637 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 logging 2013-04-03 10:34:09.939432 mon.0 10.252.24.25:6789/0 19 : [INF] pgmap v28967: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 516B/s wr, 0op/s -22473> 2013-04-03 10:34:09.993663 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112916 ==== paxos(lease_ack lc 427210 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (1510689869 0 0) 0x219c780 con 0x208a840 -22472> 2013-04-03 10:34:09.993744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos -22471> 2013-04-03 10:34:09.993764 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393972 update_from_paxos version 393972 summary v 393972 -22470> 2013-04-03 10:34:09.993781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22469> 2013-04-03 10:34:10.582666 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112917 ==== forward(pg_stats(5 pgs tid 3 v 599) v1) to leader v1 ==== 2249+0+0 (2700372181 0 0) 0x1bae8c0 con 0x208a840 -22468> 2013-04-03 10:34:10.582686 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -22467> 2013-04-03 10:34:10.582696 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad240 from 10.252.24.88:6789/0 -22466> 2013-04-03 10:34:10.582746 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 preprocess_query pg_stats(5 pgs tid 3 v 599) v1 from osd.1 10.252.24.25:6805/3572 -22465> 2013-04-03 10:34:10.582753 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 prepare_update pg_stats(5 pgs tid 3 v 599) v1 from osd.1 10.252.24.25:6805/3572 -22464> 2013-04-03 10:34:10.582757 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 prepare_pg_stats pg_stats(5 pgs tid 3 v 599) v1 from osd.1 -22463> 2013-04-03 10:34:10.582763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 got osd.1 osd_stat(1113 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1113 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -22462> 2013-04-03 10:34:10.733743 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112918 ==== forward(pg_stats(4 pgs tid 30431 v 599) v1) to leader v1 ==== 1876+0+0 (352303253 0 0) 0x1baeb80 con 0x208a840 -22461> 2013-04-03 10:34:10.733762 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -22460> 2013-04-03 10:34:10.733771 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fcd80 from 10.252.24.88:6789/0 -22459> 2013-04-03 10:34:10.733819 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 preprocess_query pg_stats(4 pgs tid 30431 v 599) v1 from osd.3 10.252.24.88:6801/17538 -22458> 2013-04-03 10:34:10.733827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 prepare_update pg_stats(4 pgs tid 30431 v 599) v1 from osd.3 10.252.24.88:6801/17538 -22457> 2013-04-03 10:34:10.733831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 prepare_pg_stats pg_stats(4 pgs tid 30431 v 599) v1 from osd.3 -22456> 2013-04-03 10:34:10.733836 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28967 got osd.3 osd_stat(1191 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1190 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -22455> 2013-04-03 10:34:10.767818 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112919 ==== forward(mdsbeacon(11234/hotcpc9039 up:replay seq 8 v184) v2) to leader v1 ==== 560+0+0 (3132721914 0 0) 0x1bae600 con 0x208a840 -22454> 2013-04-03 10:34:10.767831 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -22453> 2013-04-03 10:34:10.767838 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4340 from 10.252.24.88:6789/0 -22452> 2013-04-03 10:34:10.767871 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 8 v184) v2 from mds.0 10.252.24.25:6802/3144 -22451> 2013-04-03 10:34:10.767882 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:replay seq 8 v184) v2 tid 17389) v2 -- ?+0 0x20fc240 con 0x208a840 -22450> 2013-04-03 10:34:10.938999 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393972 encode_pending v393973 -22449> 2013-04-03 10:34:10.939027 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427210) queue_proposal bl 442 bytes; ctx = 0x1b91700 -22448> 2013-04-03 10:34:10.939036 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427210) propose_queued 427211 442 bytes -22447> 2013-04-03 10:34:10.940086 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427210 fc 0 pn 17600 opn 0) v3 -- ?+0 0x219cc80 -22446> 2013-04-03 10:34:10.940159 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28967 encode_pending v 28968 -22445> 2013-04-03 10:34:10.940212 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427192..427210) queue_proposal bl 3704 bytes; ctx = 0x1b917e0 -22444> 2013-04-03 10:34:10.940219 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427192..427210) propose_new_value not active; proposal queued -22443> 2013-04-03 10:34:11.120683 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112920 ==== paxos(accept lc 427210 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3611562944 0 0) 0x219cc80 con 0x208a840 -22442> 2013-04-03 10:34:11.121710 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427211 fc 0 pn 17600 opn 0) v3 -- ?+0 0x219c780 -22441> 2013-04-03 10:34:11.121751 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427211 fc 427192 pn 0 opn 0) v3 -- ?+0 0x219c000 -22440> 2013-04-03 10:34:11.121798 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos -22439> 2013-04-03 10:34:11.121834 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos version 393973 summary v 393972 -22438> 2013-04-03 10:34:11.121848 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos latest full 393798 -22437> 2013-04-03 10:34:11.121865 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393973 update_from_paxos applying incremental log 393973 2013-04-03 10:34:09.939432 mon.0 10.252.24.25:6789/0 19 : [INF] pgmap v28967: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 516B/s wr, 0op/s -22436> 2013-04-03 10:34:11.121907 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 check_subs -22435> 2013-04-03 10:34:11.121936 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 create_pending v 393974 -22434> 2013-04-03 10:34:11.121957 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393973 _updated_log for mon.0 10.252.24.25:6789/0 -22433> 2013-04-03 10:34:11.121970 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 19) v1 -- ?+0 0x2294540 con 0x1bac160 -22432> 2013-04-03 10:34:11.121994 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427211) propose_queued 427212 3704 bytes -22431> 2013-04-03 10:34:11.123156 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427211 fc 0 pn 17600 opn 0) v3 -- ?+0 0x219c500 -22430> 2013-04-03 10:34:11.123192 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 19) v1 ==== 0+0+0 (0 0 0) 0x2294540 con 0x1bac160 -22429> 2013-04-03 10:34:11.123203 7fcdffcdb700 10 handle_log_ack log(last 19) v1 -22428> 2013-04-03 10:34:11.123206 7fcdffcdb700 10 logged 2013-04-03 10:34:09.939432 mon.0 10.252.24.25:6789/0 19 : [INF] pgmap v28967: 240 pgs: 240 active+clean; 258 MB data, 5677 MB used, 1067 GB / 1079 GB avail; 516B/s wr, 0op/s -22427> 2013-04-03 10:34:11.174499 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112921 ==== paxos(lease_ack lc 427211 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (3969062618 0 0) 0x219c500 con 0x208a840 -22426> 2013-04-03 10:34:11.200993 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112922 ==== paxos(accept lc 427211 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4182369493 0 0) 0x219c000 con 0x208a840 -22425> 2013-04-03 10:34:11.201853 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427212 fc 0 pn 17600 opn 0) v3 -- ?+0 0x219c500 -22424> 2013-04-03 10:34:11.201880 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427212 fc 427192 pn 0 opn 0) v3 -- ?+0 0x219cc80 -22423> 2013-04-03 10:34:11.201962 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28967 update_from_paxos applying incremental 28968 -22422> 2013-04-03 10:34:11.202003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 v28968: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail -22421> 2013-04-03 10:34:11.202023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 send_pg_creates to 0 pgs -22420> 2013-04-03 10:34:11.202027 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 update_logger -22419> 2013-04-03 10:34:11.202039 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 create_pending v 28969 -22418> 2013-04-03 10:34:11.202047 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28968 _updated_stats for osd.1 10.252.24.25:6805/3572 -22417> 2013-04-03 10:34:11.202054 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(5 pgs tid 3) v1 tid 17387) v2 -- ?+0 0x20ad900 con 0x208a840 -22416> 2013-04-03 10:34:11.202065 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28968 _updated_stats for osd.3 10.252.24.88:6801/17538 -22415> 2013-04-03 10:34:11.202070 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(4 pgs tid 30431) v1 tid 17388) v2 -- ?+0 0x20ad240 con 0x208a840 -22414> 2013-04-03 10:34:11.202104 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 check_osd_map already seen 599 >= 599 -22413> 2013-04-03 10:34:11.202108 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 update_logger -22412> 2013-04-03 10:34:11.202119 7fcdffcdb700 0 log [INF] : pgmap v28968: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail -22411> 2013-04-03 10:34:11.202120 7fcdffcdb700 10 send_log to self -22410> 2013-04-03 10:34:11.202121 7fcdffcdb700 10 log_queue is 1 last_log 20 sent 19 num 1 unsent 1 sending 1 -22409> 2013-04-03 10:34:11.202123 7fcdffcdb700 10 will send 2013-04-03 10:34:11.202120 mon.0 10.252.24.25:6789/0 20 : [INF] pgmap v28968: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail -22408> 2013-04-03 10:34:11.202129 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x1c74a00 -22407> 2013-04-03 10:34:11.202176 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos -22406> 2013-04-03 10:34:11.202193 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos version 393973 summary v 393973 -22405> 2013-04-03 10:34:11.202213 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22404> 2013-04-03 10:34:11.202233 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x1c74a00 con 0x1bac160 -22403> 2013-04-03 10:34:11.202254 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos -22402> 2013-04-03 10:34:11.202268 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos version 393973 summary v 393973 -22401> 2013-04-03 10:34:11.202277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -22400> 2013-04-03 10:34:11.202288 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 preprocess_log log(1 entries) v1 from mon.0 -22399> 2013-04-03 10:34:11.202308 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -22398> 2013-04-03 10:34:11.202321 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 prepare_log log(1 entries) v1 from mon.0 -22397> 2013-04-03 10:34:11.202331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 logging 2013-04-03 10:34:11.202120 mon.0 10.252.24.25:6789/0 20 : [INF] pgmap v28968: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail -22396> 2013-04-03 10:34:11.252111 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112923 ==== paxos(lease_ack lc 427212 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (1216296095 0 0) 0x219cc80 con 0x208a840 -22395> 2013-04-03 10:34:11.252151 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos -22394> 2013-04-03 10:34:11.252167 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393973 update_from_paxos version 393973 summary v 393973 -22393> 2013-04-03 10:34:11.252181 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22392> 2013-04-03 10:34:11.586441 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 8 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140296 v184) v2 ==== 289+0+0 (2402640111 0 0) 0x21452c0 con 0x2118420 -22391> 2013-04-03 10:34:11.586508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140296 v184) v2 from mds.? 10.252.58.31:6800/10908 -22390> 2013-04-03 10:34:11.586521 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140296 v184) v2 -- ?+0 0x1bae600 con 0x2118420 -22389> 2013-04-03 10:34:12.201180 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393973 encode_pending v393974 -22388> 2013-04-03 10:34:12.201720 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427212) queue_proposal bl 424 bytes; ctx = 0x1b91780 -22387> 2013-04-03 10:34:12.201728 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427212) propose_queued 427213 424 bytes -22386> 2013-04-03 10:34:12.202812 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427212 fc 0 pn 17600 opn 0) v3 -- ?+0 0x219ca00 -22385> 2013-04-03 10:34:12.368612 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112924 ==== paxos(accept lc 427212 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3447062366 0 0) 0x219ca00 con 0x208a840 -22384> 2013-04-03 10:34:12.369577 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427213 fc 0 pn 17600 opn 0) v3 -- ?+0 0x219cc80 -22383> 2013-04-03 10:34:12.369595 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427213 fc 427192 pn 0 opn 0) v3 -- ?+0 0x219c000 -22382> 2013-04-03 10:34:12.369635 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos -22381> 2013-04-03 10:34:12.369662 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393973 -22380> 2013-04-03 10:34:12.369675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos latest full 393798 -22379> 2013-04-03 10:34:12.369693 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393974 update_from_paxos applying incremental log 393974 2013-04-03 10:34:11.202120 mon.0 10.252.24.25:6789/0 20 : [INF] pgmap v28968: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail -22378> 2013-04-03 10:34:12.369736 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 check_subs -22377> 2013-04-03 10:34:12.369765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 create_pending v 393975 -22376> 2013-04-03 10:34:12.369786 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393974 _updated_log for mon.0 10.252.24.25:6789/0 -22375> 2013-04-03 10:34:12.369799 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 20) v1 -- ?+0 0x2294540 con 0x1bac160 -22374> 2013-04-03 10:34:12.369847 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos -22373> 2013-04-03 10:34:12.369862 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393974 -22372> 2013-04-03 10:34:12.369879 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22371> 2013-04-03 10:34:12.369899 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 20) v1 ==== 0+0+0 (0 0 0) 0x2294540 con 0x1bac160 -22370> 2013-04-03 10:34:12.369907 7fcdffcdb700 10 handle_log_ack log(last 20) v1 -22369> 2013-04-03 10:34:12.369909 7fcdffcdb700 10 logged 2013-04-03 10:34:11.202120 mon.0 10.252.24.25:6789/0 20 : [INF] pgmap v28968: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail -22368> 2013-04-03 10:34:12.406275 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112925 ==== paxos(lease_ack lc 427213 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (2025718553 0 0) 0x219c000 con 0x208a840 -22367> 2013-04-03 10:34:12.406317 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos -22366> 2013-04-03 10:34:12.406333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393974 -22365> 2013-04-03 10:34:12.406347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22364> 2013-04-03 10:34:13.321873 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28968 check_down_pgs -22363> 2013-04-03 10:34:13.321897 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28968 v28968: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail -22362> 2013-04-03 10:34:13.321922 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e184 e184: 1/1/1 up {0=hotcpc9039=up:replay}, 2 up:standby -22361> 2013-04-03 10:34:13.321947 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in -22360> 2013-04-03 10:34:13.321984 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 -22359> 2013-04-03 10:34:13.322013 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos -22358> 2013-04-03 10:34:13.322030 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393974 -22357> 2013-04-03 10:34:13.322039 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393974 log -22356> 2013-04-03 10:34:13.322058 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22355> 2013-04-03 10:34:13.322076 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -22354> 2013-04-03 10:34:13.386166 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 8 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21667 v184) v2 ==== 289+0+0 (1288738872 0 0) 0x2145b00 con 0x200d420 -22353> 2013-04-03 10:34:13.386220 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21667 v184) v2 from mds.? 10.252.24.88:6800/17213 -22352> 2013-04-03 10:34:13.386235 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21667 v184) v2 -- ?+0 0x21452c0 con 0x200d420 -22351> 2013-04-03 10:34:14.780181 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112926 ==== forward(mdsbeacon(11234/hotcpc9039 up:replay seq 9 v184) v2) to leader v1 ==== 560+0+0 (735626909 0 0) 0x1baeb80 con 0x208a840 -22350> 2013-04-03 10:34:14.780202 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -22349> 2013-04-03 10:34:14.780212 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e48c0 from 10.252.24.88:6789/0 -22348> 2013-04-03 10:34:14.780261 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:replay seq 9 v184) v2 from mds.0 10.252.24.25:6802/3144 -22347> 2013-04-03 10:34:14.780273 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:replay seq 9 v184) v2 tid 17390) v2 -- ?+0 0x20fcd80 con 0x208a840 -22346> 2013-04-03 10:34:14.810242 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112927 ==== forward(pg_stats(3 pgs tid 404 v 599) v1) to leader v1 ==== 1503+0+0 (1608190933 0 0) 0x2145b00 con 0x208a840 -22345> 2013-04-03 10:34:14.810254 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -22344> 2013-04-03 10:34:14.810260 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fcd80 from 10.252.24.88:6789/0 -22343> 2013-04-03 10:34:14.810292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 preprocess_query pg_stats(3 pgs tid 404 v 599) v1 from osd.0 10.252.58.31:6802/2682 -22342> 2013-04-03 10:34:14.810299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 prepare_update pg_stats(3 pgs tid 404 v 599) v1 from osd.0 10.252.58.31:6802/2682 -22341> 2013-04-03 10:34:14.810303 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 prepare_pg_stats pg_stats(3 pgs tid 404 v 599) v1 from osd.0 -22340> 2013-04-03 10:34:14.810309 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 got osd.0 osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -22339> 2013-04-03 10:34:14.845262 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112928 ==== forward(pg_stats(7 pgs tid 524 v 599) v1) to leader v1 ==== 2991+0+0 (2991826215 0 0) 0x20e3dc0 con 0x208a840 -22338> 2013-04-03 10:34:14.845275 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 -22337> 2013-04-03 10:34:14.845282 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad900 from 10.252.24.88:6789/0 -22336> 2013-04-03 10:34:14.845319 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 preprocess_query pg_stats(7 pgs tid 524 v 599) v1 from osd.2 10.252.24.30:6801/10373 -22335> 2013-04-03 10:34:14.845327 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 prepare_update pg_stats(7 pgs tid 524 v 599) v1 from osd.2 10.252.24.30:6801/10373 -22334> 2013-04-03 10:34:14.845330 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 prepare_pg_stats pg_stats(7 pgs tid 524 v 599) v1 from osd.2 -22333> 2013-04-03 10:34:14.845336 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28968 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[])) -22332> 2013-04-03 10:34:14.860473 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28968 encode_pending v 28969 -22331> 2013-04-03 10:34:14.860530 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427213) queue_proposal bl 4073 bytes; ctx = 0x1b91790 -22330> 2013-04-03 10:34:14.860540 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427213) propose_queued 427214 4073 bytes -22329> 2013-04-03 10:34:14.862100 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427213 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2808c80 -22328> 2013-04-03 10:34:15.025654 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112929 ==== paxos(accept lc 427213 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2213032186 0 0) 0x2808c80 con 0x208a840 -22327> 2013-04-03 10:34:15.027167 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427214 fc 0 pn 17600 opn 0) v3 -- ?+0 0x219c000 -22326> 2013-04-03 10:34:15.027202 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427214 fc 427192 pn 0 opn 0) v3 -- ?+0 0x219ca00 -22325> 2013-04-03 10:34:15.027294 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28968 update_from_paxos applying incremental 28969 -22324> 2013-04-03 10:34:15.027338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 v28969: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2456B/s wr, 0op/s -22323> 2013-04-03 10:34:15.027359 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 send_pg_creates to 0 pgs -22322> 2013-04-03 10:34:15.027363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 update_logger -22321> 2013-04-03 10:34:15.027375 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 create_pending v 28970 -22320> 2013-04-03 10:34:15.027384 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28969 _updated_stats for osd.0 10.252.58.31:6802/2682 -22319> 2013-04-03 10:34:15.027391 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(3 pgs tid 404) v1 tid 17391) v2 -- ?+0 0x20ad240 con 0x208a840 -22318> 2013-04-03 10:34:15.027401 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28969 _updated_stats for osd.2 10.252.24.30:6801/10373 -22317> 2013-04-03 10:34:15.027406 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(7 pgs tid 524) v1 tid 17392) v2 -- ?+0 0x20fcd80 con 0x208a840 -22316> 2013-04-03 10:34:15.027435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 check_osd_map already seen 599 >= 599 -22315> 2013-04-03 10:34:15.027439 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 update_logger -22314> 2013-04-03 10:34:15.027450 7fcdffcdb700 0 log [INF] : pgmap v28969: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2456B/s wr, 0op/s -22313> 2013-04-03 10:34:15.027457 7fcdffcdb700 10 send_log to self -22312> 2013-04-03 10:34:15.027459 7fcdffcdb700 10 log_queue is 1 last_log 21 sent 20 num 1 unsent 1 sending 1 -22311> 2013-04-03 10:34:15.027461 7fcdffcdb700 10 will send 2013-04-03 10:34:15.027457 mon.0 10.252.24.25:6789/0 21 : [INF] pgmap v28969: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2456B/s wr, 0op/s -22310> 2013-04-03 10:34:15.027470 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24e8c00 -22309> 2013-04-03 10:34:15.027519 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos -22308> 2013-04-03 10:34:15.027536 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393974 -22307> 2013-04-03 10:34:15.027554 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22306> 2013-04-03 10:34:15.027575 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24e8c00 con 0x1bac160 -22305> 2013-04-03 10:34:15.027597 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos -22304> 2013-04-03 10:34:15.027611 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393974 -22303> 2013-04-03 10:34:15.027620 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -22302> 2013-04-03 10:34:15.027632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 preprocess_log log(1 entries) v1 from mon.0 -22301> 2013-04-03 10:34:15.027653 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -22300> 2013-04-03 10:34:15.027665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 prepare_log log(1 entries) v1 from mon.0 -22299> 2013-04-03 10:34:15.027675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:15.027457 mon.0 10.252.24.25:6789/0 21 : [INF] pgmap v28969: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2456B/s wr, 0op/s -22298> 2013-04-03 10:34:15.243279 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112930 ==== paxos(lease_ack lc 427214 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (1018647772 0 0) 0x2041280 con 0x208a840 -22297> 2013-04-03 10:34:15.243333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos -22296> 2013-04-03 10:34:15.243350 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393974 -22295> 2013-04-03 10:34:15.243364 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22294> 2013-04-03 10:34:15.568458 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112931 ==== forward(pg_stats(1 pgs tid 4 v 599) v1) to leader v1 ==== 757+0+0 (2418102888 0 0) 0x20e3b00 con 0x208a840 -22293> 2013-04-03 10:34:15.568487 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -22292> 2013-04-03 10:34:15.568497 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fcd80 from 10.252.24.88:6789/0 -22291> 2013-04-03 10:34:15.568549 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 preprocess_query pg_stats(1 pgs tid 4 v 599) v1 from osd.1 10.252.24.25:6805/3572 -22290> 2013-04-03 10:34:15.568556 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 prepare_update pg_stats(1 pgs tid 4 v 599) v1 from osd.1 10.252.24.25:6805/3572 -22289> 2013-04-03 10:34:15.568560 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 prepare_pg_stats pg_stats(1 pgs tid 4 v 599) v1 from osd.1 -22288> 2013-04-03 10:34:15.568567 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 got osd.1 osd_stat(1113 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1113 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -22287> 2013-04-03 10:34:15.591925 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 9 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140297 v184) v2 ==== 289+0+0 (2891702162 0 0) 0x2145000 con 0x2118420 -22286> 2013-04-03 10:34:15.591971 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140297 v184) v2 from mds.? 10.252.58.31:6800/10908 -22285> 2013-04-03 10:34:15.591981 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140297 v184) v2 -- ?+0 0x20e3b00 con 0x2118420 -22284> 2013-04-03 10:34:15.738052 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112932 ==== forward(pg_stats(2 pgs tid 30432 v 599) v1) to leader v1 ==== 1130+0+0 (740668724 0 0) 0x20e3840 con 0x208a840 -22283> 2013-04-03 10:34:15.738075 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -22282> 2013-04-03 10:34:15.738084 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad240 from 10.252.24.88:6789/0 -22281> 2013-04-03 10:34:15.738132 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 preprocess_query pg_stats(2 pgs tid 30432 v 599) v1 from osd.3 10.252.24.88:6801/17538 -22280> 2013-04-03 10:34:15.738140 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 prepare_update pg_stats(2 pgs tid 30432 v 599) v1 from osd.3 10.252.24.88:6801/17538 -22279> 2013-04-03 10:34:15.738144 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 prepare_pg_stats pg_stats(2 pgs tid 30432 v 599) v1 from osd.3 -22278> 2013-04-03 10:34:15.738150 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28969 got osd.3 osd_stat(1191 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1191 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -22277> 2013-04-03 10:34:15.784614 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112933 ==== forward(log(23 entries) v1) to leader v1 ==== 6253+0+0 (794662480 0 0) 0x20e3580 con 0x208a840 -22276> 2013-04-03 10:34:15.784627 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -22275> 2013-04-03 10:34:15.784636 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24e9200 from 10.252.24.88:6789/0 -22274> 2013-04-03 10:34:15.784660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos -22273> 2013-04-03 10:34:15.784675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 update_from_paxos version 393974 summary v 393974 -22272> 2013-04-03 10:34:15.784686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 preprocess_query log(23 entries) v1 from mds.0 10.252.24.25:6802/3144 -22271> 2013-04-03 10:34:15.784697 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 preprocess_log log(23 entries) v1 from mds.0 -22270> 2013-04-03 10:34:15.784726 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 prepare_update log(23 entries) v1 from mds.0 10.252.24.25:6802/3144 -22269> 2013-04-03 10:34:15.784738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 prepare_log log(23 entries) v1 from mds.0 -22268> 2013-04-03 10:34:15.784748 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.962981 mds.0 10.252.24.25:6802/3144 1 : [WRN] replayed op client.10161:60,59 used ino 10000008c2b but session next is 1000000841c -22267> 2013-04-03 10:34:15.784763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.963022 mds.0 10.252.24.25:6802/3144 2 : [WRN] replayed op client.10161:62,59 used ino 10000008c2c but session next is 1000000841c -22266> 2013-04-03 10:34:15.784777 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.963132 mds.0 10.252.24.25:6802/3144 3 : [WRN] replayed op client.10161:66,59 used ino 10000008c2d but session next is 1000000841c -22265> 2013-04-03 10:34:15.784789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.963171 mds.0 10.252.24.25:6802/3144 4 : [WRN] replayed op client.10161:67,59 used ino 10000008c2e but session next is 1000000841c -22264> 2013-04-03 10:34:15.784804 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.963212 mds.0 10.252.24.25:6802/3144 5 : [WRN] replayed op client.10161:69,59 used ino 10000008c2f but session next is 1000000841c -22263> 2013-04-03 10:34:15.784819 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966144 mds.0 10.252.24.25:6802/3144 6 : [WRN] replayed op client.10004:112,111 used ino 10000008c3f but session next is 1000000883d -22262> 2013-04-03 10:34:15.784834 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966184 mds.0 10.252.24.25:6802/3144 7 : [WRN] replayed op client.10004:114,111 used ino 10000008c40 but session next is 1000000883d -22261> 2013-04-03 10:34:15.784849 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966293 mds.0 10.252.24.25:6802/3144 8 : [WRN] replayed op client.10004:118,111 used ino 10000008c41 but session next is 1000000883d -22260> 2013-04-03 10:34:15.784865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966333 mds.0 10.252.24.25:6802/3144 9 : [WRN] replayed op client.10004:119,111 used ino 10000008c42 but session next is 1000000883d -22259> 2013-04-03 10:34:15.784880 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966373 mds.0 10.252.24.25:6802/3144 10 : [WRN] replayed op client.10004:121,111 used ino 10000008c43 but session next is 1000000883d -22258> 2013-04-03 10:34:15.784897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966759 mds.0 10.252.24.25:6802/3144 11 : [WRN] replayed op client.10004:137,136 used ino 1000000883e but session next is 10000008428 -22257> 2013-04-03 10:34:15.784913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966796 mds.0 10.252.24.25:6802/3144 12 : [WRN] replayed op client.10004:139,136 used ino 1000000883f but session next is 10000008428 -22256> 2013-04-03 10:34:15.784929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966903 mds.0 10.252.24.25:6802/3144 13 : [WRN] replayed op client.10004:143,136 used ino 10000008840 but session next is 10000008428 -22255> 2013-04-03 10:34:15.784945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966943 mds.0 10.252.24.25:6802/3144 14 : [WRN] replayed op client.10004:144,136 used ino 10000008841 but session next is 10000008428 -22254> 2013-04-03 10:34:15.784962 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.966983 mds.0 10.252.24.25:6802/3144 15 : [WRN] replayed op client.10004:146,136 used ino 10000008842 but session next is 10000008428 -22253> 2013-04-03 10:34:15.784979 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.968711 mds.0 10.252.24.25:6802/3144 16 : [WRN] replayed op client.10004:229,217 used ino 10000008c44 but session next is 10000008437 -22252> 2013-04-03 10:34:15.784995 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.968753 mds.0 10.252.24.25:6802/3144 17 : [WRN] replayed op client.10004:231,217 used ino 10000008c45 but session next is 10000008437 -22251> 2013-04-03 10:34:15.785012 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.973451 mds.0 10.252.24.25:6802/3144 18 : [WRN] replayed op client.10004:379,356 used ino 10000008c46 but session next is 1000000844c -22250> 2013-04-03 10:34:15.785029 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.973500 mds.0 10.252.24.25:6802/3144 19 : [WRN] replayed op client.10004:380,356 used ino 10000008c47 but session next is 1000000844c -22249> 2013-04-03 10:34:15.785045 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.973547 mds.0 10.252.24.25:6802/3144 20 : [WRN] replayed op client.10004:381,356 used ino 10000008c48 but session next is 1000000844c -22248> 2013-04-03 10:34:15.785065 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.973592 mds.0 10.252.24.25:6802/3144 21 : [WRN] replayed op client.10004:382,356 used ino 10000008c49 but session next is 1000000844c -22247> 2013-04-03 10:34:15.785084 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.994218 mds.0 10.252.24.25:6802/3144 22 : [WRN] replayed op client.10448:72,70 used ino 10000008ceb but session next is 100000088f7 -22246> 2013-04-03 10:34:15.785101 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393974 logging 2013-04-03 10:34:14.994380 mds.0 10.252.24.25:6802/3144 23 : [WRN] replayed op client.10448:88,86 used ino 100000088f8 but session next is 100000084c2 -22245> 2013-04-03 10:34:16.025863 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393974 encode_pending v393975 -22244> 2013-04-03 10:34:16.025909 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427214) queue_proposal bl 6386 bytes; ctx = 0x1b917d0 -22243> 2013-04-03 10:34:16.025918 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427214) propose_queued 427215 6386 bytes -22242> 2013-04-03 10:34:16.027298 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427214 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2808a00 -22241> 2013-04-03 10:34:16.027386 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28969 encode_pending v 28970 -22240> 2013-04-03 10:34:16.027425 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427192..427214) queue_proposal bl 1466 bytes; ctx = 0x1b91750 -22239> 2013-04-03 10:34:16.027432 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427192..427214) propose_new_value not active; proposal queued -22238> 2013-04-03 10:34:16.138777 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112934 ==== paxos(accept lc 427214 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1551806785 0 0) 0x2808a00 con 0x208a840 -22237> 2013-04-03 10:34:16.139935 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427215 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041280 -22236> 2013-04-03 10:34:16.139974 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427215 fc 427192 pn 0 opn 0) v3 -- ?+0 0x2808c80 -22235> 2013-04-03 10:34:16.140014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -22234> 2013-04-03 10:34:16.140041 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393974 -22233> 2013-04-03 10:34:16.140054 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos latest full 393798 -22232> 2013-04-03 10:34:16.140073 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.962981 mds.0 10.252.24.25:6802/3144 1 : [WRN] replayed op client.10161:60,59 used ino 10000008c2b but session next is 1000000841c -22231> 2013-04-03 10:34:16.140097 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.963022 mds.0 10.252.24.25:6802/3144 2 : [WRN] replayed op client.10161:62,59 used ino 10000008c2c but session next is 1000000841c -22230> 2013-04-03 10:34:16.140116 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.963132 mds.0 10.252.24.25:6802/3144 3 : [WRN] replayed op client.10161:66,59 used ino 10000008c2d but session next is 1000000841c -22229> 2013-04-03 10:34:16.140134 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.963171 mds.0 10.252.24.25:6802/3144 4 : [WRN] replayed op client.10161:67,59 used ino 10000008c2e but session next is 1000000841c -22228> 2013-04-03 10:34:16.140153 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.963212 mds.0 10.252.24.25:6802/3144 5 : [WRN] replayed op client.10161:69,59 used ino 10000008c2f but session next is 1000000841c -22227> 2013-04-03 10:34:16.140174 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966144 mds.0 10.252.24.25:6802/3144 6 : [WRN] replayed op client.10004:112,111 used ino 10000008c3f but session next is 1000000883d -22226> 2013-04-03 10:34:16.140197 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966184 mds.0 10.252.24.25:6802/3144 7 : [WRN] replayed op client.10004:114,111 used ino 10000008c40 but session next is 1000000883d -22225> 2013-04-03 10:34:16.140216 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966293 mds.0 10.252.24.25:6802/3144 8 : [WRN] replayed op client.10004:118,111 used ino 10000008c41 but session next is 1000000883d -22224> 2013-04-03 10:34:16.140234 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966333 mds.0 10.252.24.25:6802/3144 9 : [WRN] replayed op client.10004:119,111 used ino 10000008c42 but session next is 1000000883d -22223> 2013-04-03 10:34:16.140251 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966373 mds.0 10.252.24.25:6802/3144 10 : [WRN] replayed op client.10004:121,111 used ino 10000008c43 but session next is 1000000883d -22222> 2013-04-03 10:34:16.140270 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966759 mds.0 10.252.24.25:6802/3144 11 : [WRN] replayed op client.10004:137,136 used ino 1000000883e but session next is 10000008428 -22221> 2013-04-03 10:34:16.140288 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966796 mds.0 10.252.24.25:6802/3144 12 : [WRN] replayed op client.10004:139,136 used ino 1000000883f but session next is 10000008428 -22220> 2013-04-03 10:34:16.140305 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966903 mds.0 10.252.24.25:6802/3144 13 : [WRN] replayed op client.10004:143,136 used ino 10000008840 but session next is 10000008428 -22219> 2013-04-03 10:34:16.140323 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966943 mds.0 10.252.24.25:6802/3144 14 : [WRN] replayed op client.10004:144,136 used ino 10000008841 but session next is 10000008428 -22218> 2013-04-03 10:34:16.140341 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.966983 mds.0 10.252.24.25:6802/3144 15 : [WRN] replayed op client.10004:146,136 used ino 10000008842 but session next is 10000008428 -22217> 2013-04-03 10:34:16.140358 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.968711 mds.0 10.252.24.25:6802/3144 16 : [WRN] replayed op client.10004:229,217 used ino 10000008c44 but session next is 10000008437 -22216> 2013-04-03 10:34:16.140376 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.968753 mds.0 10.252.24.25:6802/3144 17 : [WRN] replayed op client.10004:231,217 used ino 10000008c45 but session next is 10000008437 -22215> 2013-04-03 10:34:16.140394 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.973451 mds.0 10.252.24.25:6802/3144 18 : [WRN] replayed op client.10004:379,356 used ino 10000008c46 but session next is 1000000844c -22214> 2013-04-03 10:34:16.140411 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.973500 mds.0 10.252.24.25:6802/3144 19 : [WRN] replayed op client.10004:380,356 used ino 10000008c47 but session next is 1000000844c -22213> 2013-04-03 10:34:16.140429 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.973547 mds.0 10.252.24.25:6802/3144 20 : [WRN] replayed op client.10004:381,356 used ino 10000008c48 but session next is 1000000844c -22212> 2013-04-03 10:34:16.140446 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.973592 mds.0 10.252.24.25:6802/3144 21 : [WRN] replayed op client.10004:382,356 used ino 10000008c49 but session next is 1000000844c -22211> 2013-04-03 10:34:16.140464 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.994218 mds.0 10.252.24.25:6802/3144 22 : [WRN] replayed op client.10448:72,70 used ino 10000008ceb but session next is 100000088f7 -22210> 2013-04-03 10:34:16.140482 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:14.994380 mds.0 10.252.24.25:6802/3144 23 : [WRN] replayed op client.10448:88,86 used ino 100000088f8 but session next is 100000084c2 -22209> 2013-04-03 10:34:16.140503 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 update_from_paxos applying incremental log 393975 2013-04-03 10:34:15.027457 mon.0 10.252.24.25:6789/0 21 : [INF] pgmap v28969: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2456B/s wr, 0op/s -22208> 2013-04-03 10:34:16.140550 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 check_subs -22207> 2013-04-03 10:34:16.140583 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 create_pending v 393976 -22206> 2013-04-03 10:34:16.140605 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 _updated_log for mon.0 10.252.24.25:6789/0 -22205> 2013-04-03 10:34:16.140617 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 21) v1 -- ?+0 0x2294a80 con 0x1bac160 -22204> 2013-04-03 10:34:16.140625 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393975 _updated_log for mds.0 10.252.24.25:6802/3144 -22203> 2013-04-03 10:34:16.140639 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(log(last 23) v1 tid 17395) v2 -- ?+0 0x20ad900 con 0x208a840 -22202> 2013-04-03 10:34:16.140667 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427215) propose_queued 427216 1466 bytes -22201> 2013-04-03 10:34:16.141310 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427215 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041500 -22200> 2013-04-03 10:34:16.141345 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 21) v1 ==== 0+0+0 (0 0 0) 0x2294a80 con 0x1bac160 -22199> 2013-04-03 10:34:16.141352 7fcdffcdb700 10 handle_log_ack log(last 21) v1 -22198> 2013-04-03 10:34:16.141355 7fcdffcdb700 10 logged 2013-04-03 10:34:15.027457 mon.0 10.252.24.25:6789/0 21 : [INF] pgmap v28969: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2456B/s wr, 0op/s -22197> 2013-04-03 10:34:16.403656 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112935 ==== paxos(lease_ack lc 427215 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (1716635401 0 0) 0x2041500 con 0x208a840 -22196> 2013-04-03 10:34:16.497742 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112936 ==== paxos(accept lc 427215 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2332786985 0 0) 0x2808c80 con 0x208a840 -22195> 2013-04-03 10:34:16.498121 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427216 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2041500 -22194> 2013-04-03 10:34:16.498140 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427216 fc 427192 pn 0 opn 0) v3 -- ?+0 0x2808a00 -22193> 2013-04-03 10:34:16.498366 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28969 update_from_paxos applying incremental 28970 -22192> 2013-04-03 10:34:16.498392 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s -22191> 2013-04-03 10:34:16.498413 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 send_pg_creates to 0 pgs -22190> 2013-04-03 10:34:16.498416 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 update_logger -22189> 2013-04-03 10:34:16.498428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 create_pending v 28971 -22188> 2013-04-03 10:34:16.498439 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28970 _updated_stats for osd.1 10.252.24.25:6805/3572 -22187> 2013-04-03 10:34:16.498447 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 4) v1 tid 17393) v2 -- ?+0 0x20fc6c0 con 0x208a840 -22186> 2013-04-03 10:34:16.498457 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28970 _updated_stats for osd.3 10.252.24.88:6801/17538 -22185> 2013-04-03 10:34:16.498462 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(2 pgs tid 30432) v1 tid 17394) v2 -- ?+0 0x20fcd80 con 0x208a840 -22184> 2013-04-03 10:34:16.498495 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 check_osd_map already seen 599 >= 599 -22183> 2013-04-03 10:34:16.498499 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 update_logger -22182> 2013-04-03 10:34:16.498511 7fcdffcdb700 0 log [INF] : pgmap v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s -22181> 2013-04-03 10:34:16.498512 7fcdffcdb700 10 send_log to self -22180> 2013-04-03 10:34:16.498513 7fcdffcdb700 10 log_queue is 1 last_log 22 sent 21 num 1 unsent 1 sending 1 -22179> 2013-04-03 10:34:16.498515 7fcdffcdb700 10 will send 2013-04-03 10:34:16.498512 mon.0 10.252.24.25:6789/0 22 : [INF] pgmap v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s -22178> 2013-04-03 10:34:16.498522 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24e8200 -22177> 2013-04-03 10:34:16.498570 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -22176> 2013-04-03 10:34:16.498588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 -22175> 2013-04-03 10:34:16.498605 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22174> 2013-04-03 10:34:16.498625 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24e8200 con 0x1bac160 -22173> 2013-04-03 10:34:16.498646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -22172> 2013-04-03 10:34:16.498660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 -22171> 2013-04-03 10:34:16.498669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -22170> 2013-04-03 10:34:16.498685 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_log log(1 entries) v1 from mon.0 -22169> 2013-04-03 10:34:16.498702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -22168> 2013-04-03 10:34:16.498713 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_log log(1 entries) v1 from mon.0 -22167> 2013-04-03 10:34:16.498723 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.498512 mon.0 10.252.24.25:6789/0 22 : [INF] pgmap v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s -22166> 2013-04-03 10:34:16.523893 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112937 ==== paxos(lease_ack lc 427216 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (793042710 0 0) 0x2808a00 con 0x208a840 -22165> 2013-04-03 10:34:16.523927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -22164> 2013-04-03 10:34:16.523943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 -22163> 2013-04-03 10:34:16.523957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22162> 2013-04-03 10:34:16.718571 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112938 ==== forward(mdsbeacon(11234/hotcpc9039 up:reconnect seq 10 v184) v2) to leader v1 ==== 560+0+0 (4055091724 0 0) 0x20e32c0 con 0x208a840 -22161> 2013-04-03 10:34:16.718593 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -22160> 2013-04-03 10:34:16.718603 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e3000 from 10.252.24.88:6789/0 -22159> 2013-04-03 10:34:16.718651 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 preprocess_query mdsbeacon(11234/hotcpc9039 up:reconnect seq 10 v184) v2 from mds.0 10.252.24.25:6802/3144 -22158> 2013-04-03 10:34:16.718666 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e184 prepare_update mdsbeacon(11234/hotcpc9039 up:reconnect seq 10 v184) v2 -22157> 2013-04-03 10:34:16.718672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 prepare_beacon mds.0 up:replay -> up:reconnect standby_for_rank=-1 -22156> 2013-04-03 10:34:16.718674 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e184 prepare_beacon pending map now: -22155> 2013-04-03 10:34:16.718677 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e184 print_map epoch 185 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:33:42.867413 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 596 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:reconnect seq 10 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11006: 10.252.24.88:6800/17213 'hotcpc12533' mds.-1.0 up:standby seq 1 -22154> 2013-04-03 10:34:16.718711 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 encode_pending e185 -22153> 2013-04-03 10:34:16.718743 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427216) queue_proposal bl 1079 bytes; ctx = 0x1b91550 -22152> 2013-04-03 10:34:16.718752 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427216) propose_queued 427217 1079 bytes -22151> 2013-04-03 10:34:16.779399 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427216 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2808a00 -22150> 2013-04-03 10:34:16.810525 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112939 ==== paxos(accept lc 427216 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2923994682 0 0) 0x2808a00 con 0x208a840 -22149> 2013-04-03 10:34:16.820657 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427217 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2808c80 -22148> 2013-04-03 10:34:16.820675 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427217 fc 427192 pn 0 opn 0) v3 -- ?+0 0x282fc80 -22147> 2013-04-03 10:34:16.820742 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 update_from_paxos version 185, my e 184 -22146> 2013-04-03 10:34:16.820769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e184 update_from_paxos got 185 -22145> 2013-04-03 10:34:16.820784 7fcdffcdb700 4 mon.hotcpc9039@0(leader).mds e185 new map -22144> 2013-04-03 10:34:16.820787 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e185 print_map epoch 185 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:34:16.718715 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 596 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:reconnect seq 10 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11006: 10.252.24.88:6800/17213 'hotcpc12533' mds.-1.0 up:standby seq 1 -22143> 2013-04-03 10:34:16.820813 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.58.31:6800/10908 -- mdsmap(e 185) v1 -- ?+0 0x1ba3400 -22142> 2013-04-03 10:34:16.820828 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/17213 -- mdsmap(e 185) v1 -- ?+0 0x2117c00 -22141> 2013-04-03 10:34:16.820837 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 update_logger -22140> 2013-04-03 10:34:16.820859 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 create_pending e186 -22139> 2013-04-03 10:34:16.820872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 _updated mds.0 mdsbeacon(11234/hotcpc9039 up:reconnect seq 10 v184) v2 -22138> 2013-04-03 10:34:16.820882 7fcdffcdb700 0 log [INF] : mds.0 10.252.24.25:6802/3144 up:reconnect -22137> 2013-04-03 10:34:16.820884 7fcdffcdb700 10 send_log to self -22136> 2013-04-03 10:34:16.820885 7fcdffcdb700 10 log_queue is 2 last_log 23 sent 22 num 2 unsent 1 sending 1 -22135> 2013-04-03 10:34:16.820887 7fcdffcdb700 10 will send 2013-04-03 10:34:16.498512 mon.0 10.252.24.25:6789/0 22 : [INF] pgmap v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s -22134> 2013-04-03 10:34:16.820892 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24e9800 -22133> 2013-04-03 10:34:16.820910 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 e185: 1/1/1 up {0=hotcpc9039=up:reconnect}, 2 up:standby -22132> 2013-04-03 10:34:16.820923 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 update_logger -22131> 2013-04-03 10:34:16.820928 7fcdffcdb700 0 log [INF] : mdsmap e185: 1/1/1 up {0=hotcpc9039=up:reconnect}, 2 up:standby -22130> 2013-04-03 10:34:16.820929 7fcdffcdb700 10 send_log to self -22129> 2013-04-03 10:34:16.820930 7fcdffcdb700 10 log_queue is 3 last_log 24 sent 22 num 3 unsent 2 sending 2 -22128> 2013-04-03 10:34:16.820931 7fcdffcdb700 10 will send 2013-04-03 10:34:16.498512 mon.0 10.252.24.25:6789/0 22 : [INF] pgmap v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s -22127> 2013-04-03 10:34:16.820936 7fcdffcdb700 10 will send 2013-04-03 10:34:16.820883 mon.0 10.252.24.25:6789/0 23 : [INF] mds.0 10.252.24.25:6802/3144 up:reconnect -22126> 2013-04-03 10:34:16.820941 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x24e9600 -22125> 2013-04-03 10:34:16.820992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -22124> 2013-04-03 10:34:16.821010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 -22123> 2013-04-03 10:34:16.821027 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22122> 2013-04-03 10:34:16.821047 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24e9800 con 0x1bac160 -22121> 2013-04-03 10:34:16.821068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -22120> 2013-04-03 10:34:16.821082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 -22119> 2013-04-03 10:34:16.821091 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -22118> 2013-04-03 10:34:16.821103 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_log log(1 entries) v1 from mon.0 -22117> 2013-04-03 10:34:16.821120 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 nothing new -22116> 2013-04-03 10:34:16.821130 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x24e9600 con 0x1bac160 -22115> 2013-04-03 10:34:16.821147 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -22114> 2013-04-03 10:34:16.821161 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 -22113> 2013-04-03 10:34:16.821173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -22112> 2013-04-03 10:34:16.821183 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_log log(2 entries) v1 from mon.0 -22111> 2013-04-03 10:34:16.821202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -22110> 2013-04-03 10:34:16.821214 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_log log(2 entries) v1 from mon.0 -22109> 2013-04-03 10:34:16.821223 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.498512 mon.0 10.252.24.25:6789/0 22 : [INF] pgmap v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s -22108> 2013-04-03 10:34:16.821239 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.820883 mon.0 10.252.24.25:6789/0 23 : [INF] mds.0 10.252.24.25:6802/3144 up:reconnect -22107> 2013-04-03 10:34:16.832627 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112940 ==== paxos(lease_ack lc 427217 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (2627086723 0 0) 0x282fc80 con 0x208a840 -22106> 2013-04-03 10:34:16.832672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -22105> 2013-04-03 10:34:16.832689 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 -22104> 2013-04-03 10:34:16.832704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22103> 2013-04-03 10:34:16.860335 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112941 ==== forward(mdsbeacon(11234/hotcpc9039 up:rejoin seq 11 v185) v2) to leader v1 ==== 560+0+0 (2823180646 0 0) 0x21468c0 con 0x208a840 -22102> 2013-04-03 10:34:16.860349 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -22101> 2013-04-03 10:34:16.860356 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2146600 from 10.252.24.88:6789/0 -22100> 2013-04-03 10:34:16.860390 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 preprocess_query mdsbeacon(11234/hotcpc9039 up:rejoin seq 11 v185) v2 from mds.0 10.252.24.25:6802/3144 -22099> 2013-04-03 10:34:16.860399 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e185 prepare_update mdsbeacon(11234/hotcpc9039 up:rejoin seq 11 v185) v2 -22098> 2013-04-03 10:34:16.860402 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 prepare_beacon mds.0 up:reconnect -> up:rejoin standby_for_rank=-1 -22097> 2013-04-03 10:34:16.860404 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e185 prepare_beacon pending map now: -22096> 2013-04-03 10:34:16.860405 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e185 print_map epoch 186 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:34:16.718715 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 596 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:rejoin seq 11 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11006: 10.252.24.88:6800/17213 'hotcpc12533' mds.-1.0 up:standby seq 1 -22095> 2013-04-03 10:34:16.860434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 encode_pending e186 -22094> 2013-04-03 10:34:16.860471 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427217) queue_proposal bl 1079 bytes; ctx = 0x1b91560 -22093> 2013-04-03 10:34:16.860479 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427217) propose_queued 427218 1079 bytes -22092> 2013-04-03 10:34:16.869543 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427217 fc 0 pn 17600 opn 0) v3 -- ?+0 0x282fc80 -22091> 2013-04-03 10:34:16.878162 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112942 ==== paxos(accept lc 427217 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2509832756 0 0) 0x282fc80 con 0x208a840 -22090> 2013-04-03 10:34:16.888249 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427218 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2808a00 -22089> 2013-04-03 10:34:16.888289 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427218 fc 427192 pn 0 opn 0) v3 -- ?+0 0x282fa00 -22088> 2013-04-03 10:34:16.888343 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 update_from_paxos version 186, my e 185 -22087> 2013-04-03 10:34:16.888357 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e185 update_from_paxos got 186 -22086> 2013-04-03 10:34:16.888367 7fcdffcdb700 4 mon.hotcpc9039@0(leader).mds e186 new map -22085> 2013-04-03 10:34:16.888368 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e186 print_map epoch 186 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:34:16.860438 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 596 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:rejoin seq 11 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11006: 10.252.24.88:6800/17213 'hotcpc12533' mds.-1.0 up:standby seq 1 -22084> 2013-04-03 10:34:16.888393 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.58.31:6800/10908 -- mdsmap(e 186) v1 -- ?+0 0x24e8000 -22083> 2013-04-03 10:34:16.888406 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/17213 -- mdsmap(e 186) v1 -- ?+0 0x24e9e00 -22082> 2013-04-03 10:34:16.888414 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 update_logger -22081> 2013-04-03 10:34:16.888428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 create_pending e187 -22080> 2013-04-03 10:34:16.888438 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 _updated mds.0 mdsbeacon(11234/hotcpc9039 up:rejoin seq 11 v185) v2 -22079> 2013-04-03 10:34:16.888445 7fcdffcdb700 0 log [INF] : mds.0 10.252.24.25:6802/3144 up:rejoin -22078> 2013-04-03 10:34:16.888446 7fcdffcdb700 10 send_log to self -22077> 2013-04-03 10:34:16.888447 7fcdffcdb700 10 log_queue is 4 last_log 25 sent 23 num 4 unsent 2 sending 2 -22076> 2013-04-03 10:34:16.888449 7fcdffcdb700 10 will send 2013-04-03 10:34:16.820883 mon.0 10.252.24.25:6789/0 23 : [INF] mds.0 10.252.24.25:6802/3144 up:reconnect -22075> 2013-04-03 10:34:16.888455 7fcdffcdb700 10 will send 2013-04-03 10:34:16.820929 mon.0 10.252.24.25:6789/0 24 : [INF] mdsmap e185: 1/1/1 up {0=hotcpc9039=up:reconnect}, 2 up:standby -22074> 2013-04-03 10:34:16.888459 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x24e8400 -22073> 2013-04-03 10:34:16.888475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 e186: 1/1/1 up {0=hotcpc9039=up:rejoin}, 2 up:standby -22072> 2013-04-03 10:34:16.888484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 update_logger -22071> 2013-04-03 10:34:16.888490 7fcdffcdb700 0 log [INF] : mdsmap e186: 1/1/1 up {0=hotcpc9039=up:rejoin}, 2 up:standby -22070> 2013-04-03 10:34:16.888491 7fcdffcdb700 10 send_log to self -22069> 2013-04-03 10:34:16.888491 7fcdffcdb700 10 log_queue is 5 last_log 26 sent 24 num 5 unsent 2 sending 2 -22068> 2013-04-03 10:34:16.888493 7fcdffcdb700 10 will send 2013-04-03 10:34:16.820929 mon.0 10.252.24.25:6789/0 24 : [INF] mdsmap e185: 1/1/1 up {0=hotcpc9039=up:reconnect}, 2 up:standby -22067> 2013-04-03 10:34:16.888499 7fcdffcdb700 10 will send 2013-04-03 10:34:16.888446 mon.0 10.252.24.25:6789/0 25 : [INF] mds.0 10.252.24.25:6802/3144 up:rejoin -22066> 2013-04-03 10:34:16.888502 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x24e9800 -22065> 2013-04-03 10:34:16.888543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -22064> 2013-04-03 10:34:16.888559 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 -22063> 2013-04-03 10:34:16.888575 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22062> 2013-04-03 10:34:16.888593 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x24e8400 con 0x1bac160 -22061> 2013-04-03 10:34:16.888613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -22060> 2013-04-03 10:34:16.888628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 -22059> 2013-04-03 10:34:16.888636 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -22058> 2013-04-03 10:34:16.888646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_log log(2 entries) v1 from mon.0 -22057> 2013-04-03 10:34:16.888666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -22056> 2013-04-03 10:34:16.888682 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_log log(2 entries) v1 from mon.0 -22055> 2013-04-03 10:34:16.888691 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.820883 mon.0 10.252.24.25:6789/0 23 : [INF] mds.0 10.252.24.25:6802/3144 up:reconnect -22054> 2013-04-03 10:34:16.888708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.820929 mon.0 10.252.24.25:6789/0 24 : [INF] mdsmap e185: 1/1/1 up {0=hotcpc9039=up:reconnect}, 2 up:standby -22053> 2013-04-03 10:34:16.888734 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x24e9800 con 0x1bac160 -22052> 2013-04-03 10:34:16.888754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -22051> 2013-04-03 10:34:16.888768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 -22050> 2013-04-03 10:34:16.888777 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -22049> 2013-04-03 10:34:16.888787 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_log log(2 entries) v1 from mon.0 -22048> 2013-04-03 10:34:16.888805 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -22047> 2013-04-03 10:34:16.888817 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_log log(2 entries) v1 from mon.0 -22046> 2013-04-03 10:34:16.888826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.820929 mon.0 10.252.24.25:6789/0 24 : [INF] mdsmap e185: 1/1/1 up {0=hotcpc9039=up:reconnect}, 2 up:standby -22045> 2013-04-03 10:34:16.888842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.888446 mon.0 10.252.24.25:6789/0 25 : [INF] mds.0 10.252.24.25:6802/3144 up:rejoin -22044> 2013-04-03 10:34:16.914426 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112943 ==== paxos(lease_ack lc 427218 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (655419605 0 0) 0x282fa00 con 0x208a840 -22043> 2013-04-03 10:34:16.914465 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -22042> 2013-04-03 10:34:16.914481 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 -22041> 2013-04-03 10:34:16.914495 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -22040> 2013-04-03 10:34:16.921703 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112944 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 12 v186) v2) to leader v1 ==== 560+0+0 (713172245 0 0) 0x2146340 con 0x208a840 -22039> 2013-04-03 10:34:16.921722 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -22038> 2013-04-03 10:34:16.921731 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2146080 from 10.252.24.88:6789/0 -22037> 2013-04-03 10:34:16.921780 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 12 v186) v2 from mds.0 10.252.24.25:6802/3144 -22036> 2013-04-03 10:34:16.921790 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e186 prepare_update mdsbeacon(11234/hotcpc9039 up:active seq 12 v186) v2 -22035> 2013-04-03 10:34:16.921793 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 prepare_beacon mds.0 up:rejoin -> up:active standby_for_rank=-1 -22034> 2013-04-03 10:34:16.921795 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e186 prepare_beacon pending map now: -22033> 2013-04-03 10:34:16.921796 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e186 print_map epoch 187 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:34:16.860438 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 596 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:active seq 12 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11006: 10.252.24.88:6800/17213 'hotcpc12533' mds.-1.0 up:standby seq 1 -22032> 2013-04-03 10:34:16.921825 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 encode_pending e187 -22031> 2013-04-03 10:34:16.921857 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427218) queue_proposal bl 1079 bytes; ctx = 0x1b91710 -22030> 2013-04-03 10:34:16.921865 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427218) propose_queued 427219 1079 bytes -22029> 2013-04-03 10:34:16.922151 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427218 fc 0 pn 17600 opn 0) v3 -- ?+0 0x282fa00 -22028> 2013-04-03 10:34:16.934972 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112945 ==== paxos(accept lc 427218 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (228680510 0 0) 0x282fa00 con 0x208a840 -22027> 2013-04-03 10:34:16.935275 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427219 fc 0 pn 17600 opn 0) v3 -- ?+0 0x282fc80 -22026> 2013-04-03 10:34:16.935307 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427219 fc 427192 pn 0 opn 0) v3 -- ?+0 0x282f780 -22025> 2013-04-03 10:34:16.935371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 update_from_paxos version 187, my e 186 -22024> 2013-04-03 10:34:16.935386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e186 update_from_paxos got 187 -22023> 2013-04-03 10:34:16.935397 7fcdffcdb700 4 mon.hotcpc9039@0(leader).mds e187 new map -22022> 2013-04-03 10:34:16.935400 7fcdffcdb700 7 mon.hotcpc9039@0(leader).mds e187 print_map epoch 187 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:34:16.921828 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 596 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:active seq 12 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11006: 10.252.24.88:6800/17213 'hotcpc12533' mds.-1.0 up:standby seq 1 -22021> 2013-04-03 10:34:16.935425 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.58.31:6800/10908 -- mdsmap(e 187) v1 -- ?+0 0x24e9c00 -22020> 2013-04-03 10:34:16.935439 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/17213 -- mdsmap(e 187) v1 -- ?+0 0x24e8c00 -22019> 2013-04-03 10:34:16.935449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 update_logger -22018> 2013-04-03 10:34:16.935472 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 create_pending e188 -22017> 2013-04-03 10:34:16.935484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 _updated mds.0 mdsbeacon(11234/hotcpc9039 up:active seq 12 v186) v2 -22016> 2013-04-03 10:34:16.935494 7fcdffcdb700 0 log [INF] : mds.0 10.252.24.25:6802/3144 up:active -22015> 2013-04-03 10:34:16.935496 7fcdffcdb700 10 send_log to self -22014> 2013-04-03 10:34:16.935496 7fcdffcdb700 10 log_queue is 6 last_log 27 sent 25 num 6 unsent 2 sending 2 -22013> 2013-04-03 10:34:16.935498 7fcdffcdb700 10 will send 2013-04-03 10:34:16.888446 mon.0 10.252.24.25:6789/0 25 : [INF] mds.0 10.252.24.25:6802/3144 up:rejoin -22012> 2013-04-03 10:34:16.935502 7fcdffcdb700 10 will send 2013-04-03 10:34:16.888491 mon.0 10.252.24.25:6789/0 26 : [INF] mdsmap e186: 1/1/1 up {0=hotcpc9039=up:rejoin}, 2 up:standby -22011> 2013-04-03 10:34:16.935506 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x24e9a00 -22010> 2013-04-03 10:34:16.935527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -22009> 2013-04-03 10:34:16.935537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 update_logger -22008> 2013-04-03 10:34:16.935543 7fcdffcdb700 0 log [INF] : mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -22007> 2013-04-03 10:34:16.935543 7fcdffcdb700 10 send_log to self -22006> 2013-04-03 10:34:16.935544 7fcdffcdb700 10 log_queue is 7 last_log 28 sent 26 num 7 unsent 2 sending 2 -22005> 2013-04-03 10:34:16.935545 7fcdffcdb700 10 will send 2013-04-03 10:34:16.888491 mon.0 10.252.24.25:6789/0 26 : [INF] mdsmap e186: 1/1/1 up {0=hotcpc9039=up:rejoin}, 2 up:standby -22004> 2013-04-03 10:34:16.935550 7fcdffcdb700 10 will send 2013-04-03 10:34:16.935495 mon.0 10.252.24.25:6789/0 27 : [INF] mds.0 10.252.24.25:6802/3144 up:active -22003> 2013-04-03 10:34:16.935553 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x24e9200 -22002> 2013-04-03 10:34:16.935596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -22001> 2013-04-03 10:34:16.935613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 -22000> 2013-04-03 10:34:16.935631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21999> 2013-04-03 10:34:16.935652 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x24e9a00 con 0x1bac160 -21998> 2013-04-03 10:34:16.935673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -21997> 2013-04-03 10:34:16.935687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 -21996> 2013-04-03 10:34:16.935696 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -21995> 2013-04-03 10:34:16.935706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_log log(2 entries) v1 from mon.0 -21994> 2013-04-03 10:34:16.935726 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -21993> 2013-04-03 10:34:16.935737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_log log(2 entries) v1 from mon.0 -21992> 2013-04-03 10:34:16.935747 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.888446 mon.0 10.252.24.25:6789/0 25 : [INF] mds.0 10.252.24.25:6802/3144 up:rejoin -21991> 2013-04-03 10:34:16.935763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.888491 mon.0 10.252.24.25:6789/0 26 : [INF] mdsmap e186: 1/1/1 up {0=hotcpc9039=up:rejoin}, 2 up:standby -21990> 2013-04-03 10:34:16.935790 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x24e9200 con 0x1bac160 -21989> 2013-04-03 10:34:16.935810 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -21988> 2013-04-03 10:34:16.935824 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 -21987> 2013-04-03 10:34:16.935832 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -21986> 2013-04-03 10:34:16.935842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 preprocess_log log(2 entries) v1 from mon.0 -21985> 2013-04-03 10:34:16.935860 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -21984> 2013-04-03 10:34:16.935872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 prepare_log log(2 entries) v1 from mon.0 -21983> 2013-04-03 10:34:16.935881 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.888491 mon.0 10.252.24.25:6789/0 26 : [INF] mdsmap e186: 1/1/1 up {0=hotcpc9039=up:rejoin}, 2 up:standby -21982> 2013-04-03 10:34:16.935897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 logging 2013-04-03 10:34:16.935495 mon.0 10.252.24.25:6789/0 27 : [INF] mds.0 10.252.24.25:6802/3144 up:active -21981> 2013-04-03 10:34:16.961098 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112946 ==== paxos(lease_ack lc 427219 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (1024342661 0 0) 0x282f780 con 0x208a840 -21980> 2013-04-03 10:34:16.961137 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos -21979> 2013-04-03 10:34:16.961153 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393975 update_from_paxos version 393975 summary v 393975 -21978> 2013-04-03 10:34:16.961167 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21977> 2013-04-03 10:34:17.376254 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 9 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21668 v187) v2 ==== 289+0+0 (1642600903 0 0) 0x2145840 con 0x200d420 -21976> 2013-04-03 10:34:17.376325 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21668 v187) v2 from mds.? 10.252.24.88:6800/17213 -21975> 2013-04-03 10:34:17.376338 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21668 v187) v2 -- ?+0 0x2146080 con 0x200d420 -21974> 2013-04-03 10:34:17.497930 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393975 encode_pending v393976 -21973> 2013-04-03 10:34:17.497966 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427219) queue_proposal bl 1548 bytes; ctx = 0x1b917c0 -21972> 2013-04-03 10:34:17.497975 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427219) propose_queued 427220 1548 bytes -21971> 2013-04-03 10:34:17.499209 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427219 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2808780 -21970> 2013-04-03 10:34:17.530675 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112947 ==== paxos(accept lc 427219 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1777952931 0 0) 0x2808780 con 0x208a840 -21969> 2013-04-03 10:34:17.537354 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427220 fc 0 pn 17600 opn 0) v3 -- ?+0 0x282f780 -21968> 2013-04-03 10:34:17.537380 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427220 fc 427192 pn 0 opn 0) v3 -- ?+0 0x282fa00 -21967> 2013-04-03 10:34:17.537423 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos -21966> 2013-04-03 10:34:17.537457 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos version 393976 summary v 393975 -21965> 2013-04-03 10:34:17.537471 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos latest full 393798 -21964> 2013-04-03 10:34:17.537489 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 update_from_paxos applying incremental log 393976 2013-04-03 10:34:16.498512 mon.0 10.252.24.25:6789/0 22 : [INF] pgmap v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s -21963> 2013-04-03 10:34:17.537515 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 update_from_paxos applying incremental log 393976 2013-04-03 10:34:16.820883 mon.0 10.252.24.25:6789/0 23 : [INF] mds.0 10.252.24.25:6802/3144 up:reconnect -21962> 2013-04-03 10:34:17.537536 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 update_from_paxos applying incremental log 393976 2013-04-03 10:34:16.820929 mon.0 10.252.24.25:6789/0 24 : [INF] mdsmap e185: 1/1/1 up {0=hotcpc9039=up:reconnect}, 2 up:standby -21961> 2013-04-03 10:34:17.537555 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 update_from_paxos applying incremental log 393976 2013-04-03 10:34:16.888446 mon.0 10.252.24.25:6789/0 25 : [INF] mds.0 10.252.24.25:6802/3144 up:rejoin -21960> 2013-04-03 10:34:17.537573 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 update_from_paxos applying incremental log 393976 2013-04-03 10:34:16.888491 mon.0 10.252.24.25:6789/0 26 : [INF] mdsmap e186: 1/1/1 up {0=hotcpc9039=up:rejoin}, 2 up:standby -21959> 2013-04-03 10:34:17.537592 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 update_from_paxos applying incremental log 393976 2013-04-03 10:34:16.935495 mon.0 10.252.24.25:6789/0 27 : [INF] mds.0 10.252.24.25:6802/3144 up:active -21958> 2013-04-03 10:34:17.537638 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 check_subs -21957> 2013-04-03 10:34:17.537670 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 create_pending v 393977 -21956> 2013-04-03 10:34:17.537691 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 _updated_log for mon.0 10.252.24.25:6789/0 -21955> 2013-04-03 10:34:17.537703 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 22) v1 -- ?+0 0x2294a80 con 0x1bac160 -21954> 2013-04-03 10:34:17.537710 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 _updated_log for mon.0 10.252.24.25:6789/0 -21953> 2013-04-03 10:34:17.537723 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 23) v1 -- ?+0 0x2224c40 con 0x1bac160 -21952> 2013-04-03 10:34:17.537728 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 _updated_log for mon.0 10.252.24.25:6789/0 -21951> 2013-04-03 10:34:17.537739 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 24) v1 -- ?+0 0x2224540 con 0x1bac160 -21950> 2013-04-03 10:34:17.537744 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 _updated_log for mon.0 10.252.24.25:6789/0 -21949> 2013-04-03 10:34:17.537756 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 25) v1 -- ?+0 0x2224380 con 0x1bac160 -21948> 2013-04-03 10:34:17.537760 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 _updated_log for mon.0 10.252.24.25:6789/0 -21947> 2013-04-03 10:34:17.537772 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 26) v1 -- ?+0 0x22241c0 con 0x1bac160 -21946> 2013-04-03 10:34:17.537776 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393976 _updated_log for mon.0 10.252.24.25:6789/0 -21945> 2013-04-03 10:34:17.537788 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 27) v1 -- ?+0 0x22248c0 con 0x1bac160 -21944> 2013-04-03 10:34:17.537844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos -21943> 2013-04-03 10:34:17.537860 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos version 393976 summary v 393976 -21942> 2013-04-03 10:34:17.537878 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21941> 2013-04-03 10:34:17.537898 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 22) v1 ==== 0+0+0 (0 0 0) 0x2294a80 con 0x1bac160 -21940> 2013-04-03 10:34:17.537906 7fcdffcdb700 10 handle_log_ack log(last 22) v1 -21939> 2013-04-03 10:34:17.537909 7fcdffcdb700 10 logged 2013-04-03 10:34:16.498512 mon.0 10.252.24.25:6789/0 22 : [INF] pgmap v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s -21938> 2013-04-03 10:34:17.537917 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 23) v1 ==== 0+0+0 (0 0 0) 0x2224c40 con 0x1bac160 -21937> 2013-04-03 10:34:17.537923 7fcdffcdb700 10 handle_log_ack log(last 23) v1 -21936> 2013-04-03 10:34:17.537924 7fcdffcdb700 10 logged 2013-04-03 10:34:16.820883 mon.0 10.252.24.25:6789/0 23 : [INF] mds.0 10.252.24.25:6802/3144 up:reconnect -21935> 2013-04-03 10:34:17.537929 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 24) v1 ==== 0+0+0 (0 0 0) 0x2224540 con 0x1bac160 -21934> 2013-04-03 10:34:17.537935 7fcdffcdb700 10 handle_log_ack log(last 24) v1 -21933> 2013-04-03 10:34:17.537937 7fcdffcdb700 10 logged 2013-04-03 10:34:16.820929 mon.0 10.252.24.25:6789/0 24 : [INF] mdsmap e185: 1/1/1 up {0=hotcpc9039=up:reconnect}, 2 up:standby -21932> 2013-04-03 10:34:17.537943 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 25) v1 ==== 0+0+0 (0 0 0) 0x2224380 con 0x1bac160 -21931> 2013-04-03 10:34:17.537948 7fcdffcdb700 10 handle_log_ack log(last 25) v1 -21930> 2013-04-03 10:34:17.537950 7fcdffcdb700 10 logged 2013-04-03 10:34:16.888446 mon.0 10.252.24.25:6789/0 25 : [INF] mds.0 10.252.24.25:6802/3144 up:rejoin -21929> 2013-04-03 10:34:17.537955 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 26) v1 ==== 0+0+0 (0 0 0) 0x22241c0 con 0x1bac160 -21928> 2013-04-03 10:34:17.537960 7fcdffcdb700 10 handle_log_ack log(last 26) v1 -21927> 2013-04-03 10:34:17.537962 7fcdffcdb700 10 logged 2013-04-03 10:34:16.888491 mon.0 10.252.24.25:6789/0 26 : [INF] mdsmap e186: 1/1/1 up {0=hotcpc9039=up:rejoin}, 2 up:standby -21926> 2013-04-03 10:34:17.537973 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 27) v1 ==== 0+0+0 (0 0 0) 0x22248c0 con 0x1bac160 -21925> 2013-04-03 10:34:17.537979 7fcdffcdb700 10 handle_log_ack log(last 27) v1 -21924> 2013-04-03 10:34:17.537980 7fcdffcdb700 10 logged 2013-04-03 10:34:16.935495 mon.0 10.252.24.25:6789/0 27 : [INF] mds.0 10.252.24.25:6802/3144 up:active -21923> 2013-04-03 10:34:17.565125 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112948 ==== paxos(lease_ack lc 427220 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (3699010447 0 0) 0x282fa00 con 0x208a840 -21922> 2013-04-03 10:34:17.565165 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos -21921> 2013-04-03 10:34:17.565181 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos version 393976 summary v 393976 -21920> 2013-04-03 10:34:17.565196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21919> 2013-04-03 10:34:18.322222 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28970 check_down_pgs -21918> 2013-04-03 10:34:18.322255 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28970 v28970: 240 pgs: 240 active+clean; 258 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 2415B/s wr, 0op/s -21917> 2013-04-03 10:34:18.322281 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -21916> 2013-04-03 10:34:18.322303 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in -21915> 2013-04-03 10:34:18.322340 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 -21914> 2013-04-03 10:34:18.322368 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos -21913> 2013-04-03 10:34:18.322385 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos version 393976 summary v 393976 -21912> 2013-04-03 10:34:18.322394 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393976 log -21911> 2013-04-03 10:34:18.322410 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21910> 2013-04-03 10:34:18.322428 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -21909> 2013-04-03 10:34:19.576108 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 10 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140298 v187) v2 ==== 289+0+0 (215589785 0 0) 0x2146b80 con 0x2118420 -21908> 2013-04-03 10:34:19.576177 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140298 v187) v2 from mds.? 10.252.58.31:6800/10908 -21907> 2013-04-03 10:34:19.576189 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140298 v187) v2 -- ?+0 0x2145840 con 0x2118420 -21906> 2013-04-03 10:34:19.785582 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112949 ==== forward(pg_stats(4 pgs tid 405 v 599) v1) to leader v1 ==== 1876+0+0 (1128361436 0 0) 0x2145dc0 con 0x208a840 -21905> 2013-04-03 10:34:19.785610 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -21904> 2013-04-03 10:34:19.785619 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 -21903> 2013-04-03 10:34:19.785667 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 preprocess_query pg_stats(4 pgs tid 405 v 599) v1 from osd.0 10.252.58.31:6802/2682 -21902> 2013-04-03 10:34:19.785675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 prepare_update pg_stats(4 pgs tid 405 v 599) v1 from osd.0 10.252.58.31:6802/2682 -21901> 2013-04-03 10:34:19.785679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 prepare_pg_stats pg_stats(4 pgs tid 405 v 599) v1 from osd.0 -21900> 2013-04-03 10:34:19.785685 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 got osd.0 osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -21899> 2013-04-03 10:34:19.825318 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112950 ==== forward(pg_stats(3 pgs tid 525 v 599) v1) to leader v1 ==== 1499+0+0 (92939510 0 0) 0x21452c0 con 0x208a840 -21898> 2013-04-03 10:34:19.825328 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 -21897> 2013-04-03 10:34:19.825335 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -21896> 2013-04-03 10:34:19.825359 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 preprocess_query pg_stats(3 pgs tid 525 v 599) v1 from osd.2 10.252.24.30:6801/10373 -21895> 2013-04-03 10:34:19.825366 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 prepare_update pg_stats(3 pgs tid 525 v 599) v1 from osd.2 10.252.24.30:6801/10373 -21894> 2013-04-03 10:34:19.825369 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 prepare_pg_stats pg_stats(3 pgs tid 525 v 599) v1 from osd.2 -21893> 2013-04-03 10:34:19.825372 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28970 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[])) -21892> 2013-04-03 10:34:19.835875 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28970 encode_pending v 28971 -21891> 2013-04-03 10:34:19.835919 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427220) queue_proposal bl 2954 bytes; ctx = 0x1b917b0 -21890> 2013-04-03 10:34:19.835928 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427220) propose_queued 427221 2954 bytes -21889> 2013-04-03 10:34:19.838125 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427220 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2808500 -21888> 2013-04-03 10:34:19.885955 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112951 ==== paxos(accept lc 427220 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4109263594 0 0) 0x2808500 con 0x208a840 -21887> 2013-04-03 10:34:19.886291 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427221 fc 0 pn 17600 opn 0) v3 -- ?+0 0x282fa00 -21886> 2013-04-03 10:34:19.886323 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427221 fc 427192 pn 0 opn 0) v3 -- ?+0 0x2808780 -21885> 2013-04-03 10:34:19.886394 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28970 update_from_paxos applying incremental 28971 -21884> 2013-04-03 10:34:19.886424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 v28971: 240 pgs: 240 active+clean; 256 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 823B/s wr, 1op/s -21883> 2013-04-03 10:34:19.886444 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 send_pg_creates to 0 pgs -21882> 2013-04-03 10:34:19.886448 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 update_logger -21881> 2013-04-03 10:34:19.886459 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 create_pending v 28972 -21880> 2013-04-03 10:34:19.886468 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28971 _updated_stats for osd.0 10.252.58.31:6802/2682 -21879> 2013-04-03 10:34:19.886475 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(4 pgs tid 405) v1 tid 17399) v2 -- ?+0 0x20ad900 con 0x208a840 -21878> 2013-04-03 10:34:19.886486 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28971 _updated_stats for osd.2 10.252.24.30:6801/10373 -21877> 2013-04-03 10:34:19.886490 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(3 pgs tid 525) v1 tid 17400) v2 -- ?+0 0x20fc240 con 0x208a840 -21876> 2013-04-03 10:34:19.886513 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 check_osd_map already seen 599 >= 599 -21875> 2013-04-03 10:34:19.886517 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 update_logger -21874> 2013-04-03 10:34:19.886527 7fcdffcdb700 0 log [INF] : pgmap v28971: 240 pgs: 240 active+clean; 256 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 823B/s wr, 1op/s -21873> 2013-04-03 10:34:19.886529 7fcdffcdb700 10 send_log to self -21872> 2013-04-03 10:34:19.886530 7fcdffcdb700 10 log_queue is 2 last_log 29 sent 27 num 2 unsent 2 sending 2 -21871> 2013-04-03 10:34:19.886532 7fcdffcdb700 10 will send 2013-04-03 10:34:16.935543 mon.0 10.252.24.25:6789/0 28 : [INF] mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -21870> 2013-04-03 10:34:19.886539 7fcdffcdb700 10 will send 2013-04-03 10:34:19.886529 mon.0 10.252.24.25:6789/0 29 : [INF] pgmap v28971: 240 pgs: 240 active+clean; 256 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 823B/s wr, 1op/s -21869> 2013-04-03 10:34:19.886544 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x25fb400 -21868> 2013-04-03 10:34:19.886593 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos -21867> 2013-04-03 10:34:19.886612 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos version 393976 summary v 393976 -21866> 2013-04-03 10:34:19.886630 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21865> 2013-04-03 10:34:19.886651 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x25fb400 con 0x1bac160 -21864> 2013-04-03 10:34:19.886672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos -21863> 2013-04-03 10:34:19.886686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos version 393976 summary v 393976 -21862> 2013-04-03 10:34:19.886695 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -21861> 2013-04-03 10:34:19.886705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 preprocess_log log(2 entries) v1 from mon.0 -21860> 2013-04-03 10:34:19.886727 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -21859> 2013-04-03 10:34:19.886739 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 prepare_log log(2 entries) v1 from mon.0 -21858> 2013-04-03 10:34:19.886748 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 logging 2013-04-03 10:34:16.935543 mon.0 10.252.24.25:6789/0 28 : [INF] mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -21857> 2013-04-03 10:34:19.886766 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 logging 2013-04-03 10:34:19.886529 mon.0 10.252.24.25:6789/0 29 : [INF] pgmap v28971: 240 pgs: 240 active+clean; 256 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 823B/s wr, 1op/s -21856> 2013-04-03 10:34:19.911953 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112952 ==== paxos(lease_ack lc 427221 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (3917344040 0 0) 0x2808780 con 0x208a840 -21855> 2013-04-03 10:34:19.911988 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos -21854> 2013-04-03 10:34:19.912003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393976 update_from_paxos version 393976 summary v 393976 -21853> 2013-04-03 10:34:19.912017 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21852> 2013-04-03 10:34:20.089199 7fcdfe8d5700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2116a00 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept sd=24 10.252.24.88:32939/0 -21851> 2013-04-03 10:34:20.089238 7fcdfe8d5700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2116a00 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:32939/0) -21850> 2013-04-03 10:34:20.089251 7fcdfe8d5700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2116a00 sd=24 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -21849> 2013-04-03 10:34:20.089453 7fcdfe8d5700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2116a00 sd=24 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -21848> 2013-04-03 10:34:20.554300 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112953 ==== forward(pg_stats(7 pgs tid 5 v 599) v1) to leader v1 ==== 2995+0+0 (3368196303 0 0) 0x2145580 con 0x208a840 -21847> 2013-04-03 10:34:20.554314 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -21846> 2013-04-03 10:34:20.554323 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 -21845> 2013-04-03 10:34:20.554374 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 preprocess_query pg_stats(7 pgs tid 5 v 599) v1 from osd.1 10.252.24.25:6805/3572 -21844> 2013-04-03 10:34:20.554383 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 prepare_update pg_stats(7 pgs tid 5 v 599) v1 from osd.1 10.252.24.25:6805/3572 -21843> 2013-04-03 10:34:20.554386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 prepare_pg_stats pg_stats(7 pgs tid 5 v 599) v1 from osd.1 -21842> 2013-04-03 10:34:20.554392 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 got osd.1 osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1113 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -21841> 2013-04-03 10:34:20.722214 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112954 ==== forward(pg_stats(8 pgs tid 30433 v 599) v1) to leader v1 ==== 3368+0+0 (2589001883 0 0) 0x1bae600 con 0x208a840 -21840> 2013-04-03 10:34:20.722242 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -21839> 2013-04-03 10:34:20.722252 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad900 from 10.252.24.88:6789/0 -21838> 2013-04-03 10:34:20.722302 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 preprocess_query pg_stats(8 pgs tid 30433 v 599) v1 from osd.3 10.252.24.88:6801/17538 -21837> 2013-04-03 10:34:20.722309 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 prepare_update pg_stats(8 pgs tid 30433 v 599) v1 from osd.3 10.252.24.88:6801/17538 -21836> 2013-04-03 10:34:20.722313 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 prepare_pg_stats pg_stats(8 pgs tid 30433 v 599) v1 from osd.3 -21835> 2013-04-03 10:34:20.722318 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28971 got osd.3 osd_stat(1192 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1191 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -21834> 2013-04-03 10:34:20.886141 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393976 encode_pending v393977 -21833> 2013-04-03 10:34:20.886166 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427221) queue_proposal bl 675 bytes; ctx = 0x1b916c0 -21832> 2013-04-03 10:34:20.886174 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427221) propose_queued 427222 675 bytes -21831> 2013-04-03 10:34:20.887331 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427221 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2808280 -21830> 2013-04-03 10:34:20.887417 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28971 encode_pending v 28972 -21829> 2013-04-03 10:34:20.887480 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427192..427221) queue_proposal bl 5942 bytes; ctx = 0x1b91830 -21828> 2013-04-03 10:34:20.887488 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427192..427221) propose_new_value not active; proposal queued -21827> 2013-04-03 10:34:20.920387 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112955 ==== paxos(accept lc 427221 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (489727797 0 0) 0x2808280 con 0x208a840 -21826> 2013-04-03 10:34:20.920707 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427222 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2808780 -21825> 2013-04-03 10:34:20.920740 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427222 fc 427192 pn 0 opn 0) v3 -- ?+0 0x2808500 -21824> 2013-04-03 10:34:20.920775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos -21823> 2013-04-03 10:34:20.920812 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos version 393977 summary v 393976 -21822> 2013-04-03 10:34:20.920835 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos latest full 393798 -21821> 2013-04-03 10:34:20.920854 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393977 update_from_paxos applying incremental log 393977 2013-04-03 10:34:16.935543 mon.0 10.252.24.25:6789/0 28 : [INF] mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -21820> 2013-04-03 10:34:20.920880 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393977 update_from_paxos applying incremental log 393977 2013-04-03 10:34:19.886529 mon.0 10.252.24.25:6789/0 29 : [INF] pgmap v28971: 240 pgs: 240 active+clean; 256 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 823B/s wr, 1op/s -21819> 2013-04-03 10:34:20.920921 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 check_subs -21818> 2013-04-03 10:34:20.920950 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 create_pending v 393978 -21817> 2013-04-03 10:34:20.920971 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393977 _updated_log for mon.0 10.252.24.25:6789/0 -21816> 2013-04-03 10:34:20.920984 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 29) v1 -- ?+0 0x2224c40 con 0x1bac160 -21815> 2013-04-03 10:34:20.921010 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427222) propose_queued 427223 5942 bytes -21814> 2013-04-03 10:34:20.922567 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427222 fc 0 pn 17600 opn 0) v3 -- ?+0 0x282f500 -21813> 2013-04-03 10:34:20.922604 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 29) v1 ==== 0+0+0 (0 0 0) 0x2224c40 con 0x1bac160 -21812> 2013-04-03 10:34:20.922618 7fcdffcdb700 10 handle_log_ack log(last 29) v1 -21811> 2013-04-03 10:34:20.922620 7fcdffcdb700 10 logged 2013-04-03 10:34:16.935543 mon.0 10.252.24.25:6789/0 28 : [INF] mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -21810> 2013-04-03 10:34:20.922626 7fcdffcdb700 10 logged 2013-04-03 10:34:19.886529 mon.0 10.252.24.25:6789/0 29 : [INF] pgmap v28971: 240 pgs: 240 active+clean; 256 MB data, 5678 MB used, 1067 GB / 1079 GB avail; 823B/s wr, 1op/s -21809> 2013-04-03 10:34:20.946441 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112956 ==== paxos(lease_ack lc 427222 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (1488692166 0 0) 0x282f500 con 0x208a840 -21808> 2013-04-03 10:34:20.971114 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112957 ==== paxos(accept lc 427222 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3486008451 0 0) 0x2116a00 con 0x208a840 -21807> 2013-04-03 10:34:20.972128 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427223 fc 0 pn 17600 opn 0) v3 -- ?+0 0x282f500 -21806> 2013-04-03 10:34:20.972152 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427223 fc 427192 pn 0 opn 0) v3 -- ?+0 0x2808280 -21805> 2013-04-03 10:34:20.972226 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28971 update_from_paxos applying incremental 28972 -21804> 2013-04-03 10:34:20.972275 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 v28972: 240 pgs: 240 active+clean; 252 MB data, 5675 MB used, 1067 GB / 1079 GB avail; 210B/s rd, 33079B/s wr, 13op/s -21803> 2013-04-03 10:34:20.972297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 send_pg_creates to 0 pgs -21802> 2013-04-03 10:34:20.972300 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 update_logger -21801> 2013-04-03 10:34:20.972310 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 create_pending v 28973 -21800> 2013-04-03 10:34:20.972319 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28972 _updated_stats for osd.1 10.252.24.25:6805/3572 -21799> 2013-04-03 10:34:20.972324 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(7 pgs tid 5) v1 tid 17401) v2 -- ?+0 0x20fc6c0 con 0x208a840 -21798> 2013-04-03 10:34:20.972335 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28972 _updated_stats for osd.3 10.252.24.88:6801/17538 -21797> 2013-04-03 10:34:20.972339 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(8 pgs tid 30433) v1 tid 17402) v2 -- ?+0 0x20fc240 con 0x208a840 -21796> 2013-04-03 10:34:20.972377 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 check_osd_map already seen 599 >= 599 -21795> 2013-04-03 10:34:20.972380 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 update_logger -21794> 2013-04-03 10:34:20.972392 7fcdffcdb700 0 log [INF] : pgmap v28972: 240 pgs: 240 active+clean; 252 MB data, 5675 MB used, 1067 GB / 1079 GB avail; 210B/s rd, 33079B/s wr, 13op/s -21793> 2013-04-03 10:34:20.972394 7fcdffcdb700 10 send_log to self -21792> 2013-04-03 10:34:20.972395 7fcdffcdb700 10 log_queue is 1 last_log 30 sent 29 num 1 unsent 1 sending 1 -21791> 2013-04-03 10:34:20.972397 7fcdffcdb700 10 will send 2013-04-03 10:34:20.972394 mon.0 10.252.24.25:6789/0 30 : [INF] pgmap v28972: 240 pgs: 240 active+clean; 252 MB data, 5675 MB used, 1067 GB / 1079 GB avail; 210B/s rd, 33079B/s wr, 13op/s -21790> 2013-04-03 10:34:20.972404 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x25fb200 -21789> 2013-04-03 10:34:20.972451 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos -21788> 2013-04-03 10:34:20.972467 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos version 393977 summary v 393977 -21787> 2013-04-03 10:34:20.972487 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21786> 2013-04-03 10:34:20.972507 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x25fb200 con 0x1bac160 -21785> 2013-04-03 10:34:20.972528 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos -21784> 2013-04-03 10:34:20.972545 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos version 393977 summary v 393977 -21783> 2013-04-03 10:34:20.972554 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21782> 2013-04-03 10:34:20.972564 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 preprocess_log log(1 entries) v1 from mon.0 -21781> 2013-04-03 10:34:20.972580 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21780> 2013-04-03 10:34:20.972592 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 prepare_log log(1 entries) v1 from mon.0 -21779> 2013-04-03 10:34:20.972601 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 logging 2013-04-03 10:34:20.972394 mon.0 10.252.24.25:6789/0 30 : [INF] pgmap v28972: 240 pgs: 240 active+clean; 252 MB data, 5675 MB used, 1067 GB / 1079 GB avail; 210B/s rd, 33079B/s wr, 13op/s -21778> 2013-04-03 10:34:20.997476 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112958 ==== paxos(lease_ack lc 427223 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (3455311583 0 0) 0x2808280 con 0x208a840 -21777> 2013-04-03 10:34:20.997512 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos -21776> 2013-04-03 10:34:20.997527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393977 update_from_paxos version 393977 summary v 393977 -21775> 2013-04-03 10:34:20.997543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21774> 2013-04-03 10:34:20.997578 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112959 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 13 v187) v2) to leader v1 ==== 560+0+0 (900017580 0 0) 0x1bae340 con 0x208a840 -21773> 2013-04-03 10:34:20.997588 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -21772> 2013-04-03 10:34:20.997595 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4600 from 10.252.24.88:6789/0 -21771> 2013-04-03 10:34:20.997620 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 13 v187) v2 from mds.0 10.252.24.25:6802/3144 -21770> 2013-04-03 10:34:20.997629 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 13 v187) v2 tid 17403) v2 -- ?+0 0x20ad900 con 0x208a840 -21769> 2013-04-03 10:34:21.376340 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 10 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21669 v187) v2 ==== 289+0+0 (1116120762 0 0) 0x20e4340 con 0x200d420 -21768> 2013-04-03 10:34:21.376394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21669 v187) v2 from mds.? 10.252.24.88:6800/17213 -21767> 2013-04-03 10:34:21.376404 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21669 v187) v2 -- ?+0 0x1bae340 con 0x200d420 -21766> 2013-04-03 10:34:21.971308 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393977 encode_pending v393978 -21765> 2013-04-03 10:34:21.971335 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427223) queue_proposal bl 456 bytes; ctx = 0x1b91840 -21764> 2013-04-03 10:34:21.971342 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427223) propose_queued 427224 456 bytes -21763> 2013-04-03 10:34:21.972360 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427223 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2808000 -21762> 2013-04-03 10:34:22.114030 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112960 ==== paxos(accept lc 427223 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4167264159 0 0) 0x2808000 con 0x208a840 -21761> 2013-04-03 10:34:22.114823 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427224 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2808280 -21760> 2013-04-03 10:34:22.114857 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427224 fc 427192 pn 0 opn 0) v3 -- ?+0 0x2116a00 -21759> 2013-04-03 10:34:22.114907 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos -21758> 2013-04-03 10:34:22.114943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393977 -21757> 2013-04-03 10:34:22.114957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos latest full 393798 -21756> 2013-04-03 10:34:22.114974 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393978 update_from_paxos applying incremental log 393978 2013-04-03 10:34:20.972394 mon.0 10.252.24.25:6789/0 30 : [INF] pgmap v28972: 240 pgs: 240 active+clean; 252 MB data, 5675 MB used, 1067 GB / 1079 GB avail; 210B/s rd, 33079B/s wr, 13op/s -21755> 2013-04-03 10:34:22.115029 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 check_subs -21754> 2013-04-03 10:34:22.115062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 create_pending v 393979 -21753> 2013-04-03 10:34:22.115083 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393978 _updated_log for mon.0 10.252.24.25:6789/0 -21752> 2013-04-03 10:34:22.115096 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 30) v1 -- ?+0 0x2224c40 con 0x1bac160 -21751> 2013-04-03 10:34:22.115144 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos -21750> 2013-04-03 10:34:22.115159 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 -21749> 2013-04-03 10:34:22.115176 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21748> 2013-04-03 10:34:22.115199 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 30) v1 ==== 0+0+0 (0 0 0) 0x2224c40 con 0x1bac160 -21747> 2013-04-03 10:34:22.115208 7fcdffcdb700 10 handle_log_ack log(last 30) v1 -21746> 2013-04-03 10:34:22.115210 7fcdffcdb700 10 logged 2013-04-03 10:34:20.972394 mon.0 10.252.24.25:6789/0 30 : [INF] pgmap v28972: 240 pgs: 240 active+clean; 252 MB data, 5675 MB used, 1067 GB / 1079 GB avail; 210B/s rd, 33079B/s wr, 13op/s -21745> 2013-04-03 10:34:22.139969 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112961 ==== paxos(lease_ack lc 427224 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (799777552 0 0) 0x2116a00 con 0x208a840 -21744> 2013-04-03 10:34:22.140010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos -21743> 2013-04-03 10:34:22.140026 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 -21742> 2013-04-03 10:34:22.140044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21741> 2013-04-03 10:34:23.322564 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28972 check_down_pgs -21740> 2013-04-03 10:34:23.322597 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28972 v28972: 240 pgs: 240 active+clean; 252 MB data, 5675 MB used, 1067 GB / 1079 GB avail; 210B/s rd, 33079B/s wr, 13op/s -21739> 2013-04-03 10:34:23.322624 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -21738> 2013-04-03 10:34:23.322651 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in -21737> 2013-04-03 10:34:23.322688 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 -21736> 2013-04-03 10:34:23.322717 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos -21735> 2013-04-03 10:34:23.322735 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 -21734> 2013-04-03 10:34:23.322744 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393978 log -21733> 2013-04-03 10:34:23.322760 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21732> 2013-04-03 10:34:23.322778 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -21731> 2013-04-03 10:34:23.576237 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 11 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140299 v187) v2 ==== 289+0+0 (800529636 0 0) 0x2145000 con 0x2118420 -21730> 2013-04-03 10:34:23.576306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140299 v187) v2 from mds.? 10.252.58.31:6800/10908 -21729> 2013-04-03 10:34:23.576318 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140299 v187) v2 -- ?+0 0x20e4340 con 0x2118420 -21728> 2013-04-03 10:34:24.922069 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112962 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 14 v187) v2) to leader v1 ==== 560+0+0 (465706384 0 0) 0x1bae600 con 0x208a840 -21727> 2013-04-03 10:34:24.922091 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -21726> 2013-04-03 10:34:24.922100 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4b80 from 10.252.24.88:6789/0 -21725> 2013-04-03 10:34:24.922150 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 14 v187) v2 from mds.0 10.252.24.25:6802/3144 -21724> 2013-04-03 10:34:24.922161 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 14 v187) v2 tid 17404) v2 -- ?+0 0x20fc240 con 0x208a840 -21723> 2013-04-03 10:34:25.114940 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427224 fc 427192 pn 0 opn 0) v3 -- ?+0 0x26dcc80 -21722> 2013-04-03 10:34:25.115739 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112963 ==== paxos(lease_ack lc 427224 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (3436817375 0 0) 0x26dcc80 con 0x208a840 -21721> 2013-04-03 10:34:25.115804 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos -21720> 2013-04-03 10:34:25.115822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 -21719> 2013-04-03 10:34:25.115839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21718> 2013-04-03 10:34:25.376441 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 11 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21670 v187) v2 ==== 289+0+0 (657811261 0 0) 0x20e4080 con 0x200d420 -21717> 2013-04-03 10:34:25.376497 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21670 v187) v2 from mds.? 10.252.24.88:6800/17213 -21716> 2013-04-03 10:34:25.376509 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21670 v187) v2 -- ?+0 0x1bae600 con 0x200d420 -21715> 2013-04-03 10:34:25.554202 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112964 ==== forward(pg_stats(1 pgs tid 6 v 599) v1) to leader v1 ==== 757+0+0 (1388022069 0 0) 0x2145000 con 0x208a840 -21714> 2013-04-03 10:34:25.554216 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -21713> 2013-04-03 10:34:25.554224 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 -21712> 2013-04-03 10:34:25.554258 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 preprocess_query pg_stats(1 pgs tid 6 v 599) v1 from osd.1 10.252.24.25:6805/3572 -21711> 2013-04-03 10:34:25.554266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 prepare_update pg_stats(1 pgs tid 6 v 599) v1 from osd.1 10.252.24.25:6805/3572 -21710> 2013-04-03 10:34:25.554270 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 prepare_pg_stats pg_stats(1 pgs tid 6 v 599) v1 from osd.1 -21709> 2013-04-03 10:34:25.554275 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28972 got osd.1 osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -21708> 2013-04-03 10:34:25.604453 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28972 encode_pending v 28973 -21707> 2013-04-03 10:34:25.604485 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427224) queue_proposal bl 654 bytes; ctx = 0x1b916a0 -21706> 2013-04-03 10:34:25.604494 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427224) propose_queued 427225 654 bytes -21705> 2013-04-03 10:34:25.606255 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427224 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dca00 -21704> 2013-04-03 10:34:25.760945 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112965 ==== paxos(accept lc 427224 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2077857585 0 0) 0x26dca00 con 0x208a840 -21703> 2013-04-03 10:34:25.761298 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427225 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dcc80 -21702> 2013-04-03 10:34:25.761327 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427225 fc 427192 pn 0 opn 0) v3 -- ?+0 0x2116a00 -21701> 2013-04-03 10:34:25.761415 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28972 update_from_paxos applying incremental 28973 -21700> 2013-04-03 10:34:25.761435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 v28973: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 177B/s rd, 64792B/s wr, 27op/s -21699> 2013-04-03 10:34:25.761453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 send_pg_creates to 0 pgs -21698> 2013-04-03 10:34:25.761456 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 update_logger -21697> 2013-04-03 10:34:25.761465 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 create_pending v 28974 -21696> 2013-04-03 10:34:25.761474 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28973 _updated_stats for osd.1 10.252.24.25:6805/3572 -21695> 2013-04-03 10:34:25.761482 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 6) v1 tid 17405) v2 -- ?+0 0x20ad900 con 0x208a840 -21694> 2013-04-03 10:34:25.761492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 check_osd_map already seen 599 >= 599 -21693> 2013-04-03 10:34:25.761495 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 update_logger -21692> 2013-04-03 10:34:25.761507 7fcdffcdb700 0 log [INF] : pgmap v28973: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 177B/s rd, 64792B/s wr, 27op/s -21691> 2013-04-03 10:34:25.761509 7fcdffcdb700 10 send_log to self -21690> 2013-04-03 10:34:25.761509 7fcdffcdb700 10 log_queue is 1 last_log 31 sent 30 num 1 unsent 1 sending 1 -21689> 2013-04-03 10:34:25.761518 7fcdffcdb700 10 will send 2013-04-03 10:34:25.761508 mon.0 10.252.24.25:6789/0 31 : [INF] pgmap v28973: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 177B/s rd, 64792B/s wr, 27op/s -21688> 2013-04-03 10:34:25.761526 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x25fb200 -21687> 2013-04-03 10:34:25.761576 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos -21686> 2013-04-03 10:34:25.761593 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 -21685> 2013-04-03 10:34:25.761610 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21684> 2013-04-03 10:34:25.761630 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x25fb200 con 0x1bac160 -21683> 2013-04-03 10:34:25.761650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos -21682> 2013-04-03 10:34:25.761664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 -21681> 2013-04-03 10:34:25.761673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21680> 2013-04-03 10:34:25.761683 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 preprocess_log log(1 entries) v1 from mon.0 -21679> 2013-04-03 10:34:25.761700 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21678> 2013-04-03 10:34:25.761711 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 prepare_log log(1 entries) v1 from mon.0 -21677> 2013-04-03 10:34:25.761721 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:25.761508 mon.0 10.252.24.25:6789/0 31 : [INF] pgmap v28973: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 177B/s rd, 64792B/s wr, 27op/s -21676> 2013-04-03 10:34:25.773172 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112966 ==== paxos(lease_ack lc 427225 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (2013751110 0 0) 0x2116a00 con 0x208a840 -21675> 2013-04-03 10:34:25.773210 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos -21674> 2013-04-03 10:34:25.773225 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 -21673> 2013-04-03 10:34:25.773239 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21672> 2013-04-03 10:34:25.773496 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112967 ==== forward(pg_stats(1 pgs tid 30434 v 599) v1) to leader v1 ==== 757+0+0 (1940564381 0 0) 0x21452c0 con 0x208a840 -21671> 2013-04-03 10:34:25.773508 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -21670> 2013-04-03 10:34:25.773516 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad900 from 10.252.24.88:6789/0 -21669> 2013-04-03 10:34:25.773541 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 preprocess_query pg_stats(1 pgs tid 30434 v 599) v1 from osd.3 10.252.24.88:6801/17538 -21668> 2013-04-03 10:34:25.773560 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 prepare_update pg_stats(1 pgs tid 30434 v 599) v1 from osd.3 10.252.24.88:6801/17538 -21667> 2013-04-03 10:34:25.773565 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 prepare_pg_stats pg_stats(1 pgs tid 30434 v 599) v1 from osd.3 -21666> 2013-04-03 10:34:25.773570 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28973 got osd.3 osd_stat(1192 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1192 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -21665> 2013-04-03 10:34:25.773883 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112968 ==== forward(log(10 entries) v1) to leader v1 ==== 2685+0+0 (3330533540 0 0) 0x2145dc0 con 0x208a840 -21664> 2013-04-03 10:34:25.773893 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -21663> 2013-04-03 10:34:25.773898 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1c75200 from 10.252.24.88:6789/0 -21662> 2013-04-03 10:34:25.773916 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos -21661> 2013-04-03 10:34:25.773931 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 update_from_paxos version 393978 summary v 393978 -21660> 2013-04-03 10:34:25.773940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 preprocess_query log(10 entries) v1 from mds.0 10.252.24.25:6802/3144 -21659> 2013-04-03 10:34:25.773950 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 preprocess_log log(10 entries) v1 from mds.0 -21658> 2013-04-03 10:34:25.773995 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 prepare_update log(10 entries) v1 from mds.0 10.252.24.25:6802/3144 -21657> 2013-04-03 10:34:25.774006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 prepare_log log(10 entries) v1 from mds.0 -21656> 2013-04-03 10:34:25.774016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.833513 mds.0 10.252.24.25:6802/3144 24 : [DBG] reconnect by client.11038 10.252.24.25:0/331381212 after 0.000310 -21655> 2013-04-03 10:34:25.774034 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.833818 mds.0 10.252.24.25:6802/3144 25 : [DBG] reconnect by client.11042 10.252.24.25:0/2614196857 after 0.000622 -21654> 2013-04-03 10:34:25.774052 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.835300 mds.0 10.252.24.25:6802/3144 26 : [DBG] reconnect by client.11041 10.252.24.25:0/3745741276 after 0.002096 -21653> 2013-04-03 10:34:25.774069 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.835370 mds.0 10.252.24.25:6802/3144 27 : [DBG] reconnect by client.11074 10.252.24.32:0/3871608469 after 0.002174 -21652> 2013-04-03 10:34:25.774088 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.835418 mds.0 10.252.24.25:6802/3144 28 : [DBG] reconnect by client.11054 10.252.24.30:0/2446411061 after 0.002224 -21651> 2013-04-03 10:34:25.774106 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.835463 mds.0 10.252.24.25:6802/3144 29 : [DBG] reconnect by client.11045 10.252.24.25:0/2151250553 after 0.002268 -21650> 2013-04-03 10:34:25.774124 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.835663 mds.0 10.252.24.25:6802/3144 30 : [DBG] reconnect by client.11802 10.252.24.30:0/517 after 0.002465 -21649> 2013-04-03 10:34:25.774142 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.835708 mds.0 10.252.24.25:6802/3144 31 : [DBG] reconnect by client.11071 10.252.24.32:0/1396772501 after 0.002514 -21648> 2013-04-03 10:34:25.774161 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.835801 mds.0 10.252.24.25:6802/3144 32 : [DBG] reconnect by client.11051 10.252.24.30:0/802047285 after 0.002603 -21647> 2013-04-03 10:34:25.774179 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393978 logging 2013-04-03 10:34:16.859400 mds.0 10.252.24.25:6802/3144 33 : [DBG] reconnect by client.11228 10.252.24.25:0/1941 after 0.026202 -21646> 2013-04-03 10:34:26.761168 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393978 encode_pending v393979 -21645> 2013-04-03 10:34:26.761211 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427225) queue_proposal bl 2831 bytes; ctx = 0x1b916b0 -21644> 2013-04-03 10:34:26.761219 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427225) propose_queued 427226 2831 bytes -21643> 2013-04-03 10:34:26.762486 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427225 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc780 -21642> 2013-04-03 10:34:26.762586 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28973 encode_pending v 28974 -21641> 2013-04-03 10:34:26.762621 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427192..427225) queue_proposal bl 654 bytes; ctx = 0x1b91800 -21640> 2013-04-03 10:34:26.762628 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427192..427225) propose_new_value not active; proposal queued -21639> 2013-04-03 10:34:26.798226 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112969 ==== paxos(accept lc 427225 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1529137976 0 0) 0x26dc780 con 0x208a840 -21638> 2013-04-03 10:34:26.800233 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427226 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116a00 -21637> 2013-04-03 10:34:26.800270 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427226 fc 427192 pn 0 opn 0) v3 -- ?+0 0x26dca00 -21636> 2013-04-03 10:34:26.800311 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos -21635> 2013-04-03 10:34:26.800336 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos version 393979 summary v 393978 -21634> 2013-04-03 10:34:26.800349 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos latest full 393798 -21633> 2013-04-03 10:34:26.800368 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.833513 mds.0 10.252.24.25:6802/3144 24 : [DBG] reconnect by client.11038 10.252.24.25:0/331381212 after 0.000310 -21632> 2013-04-03 10:34:26.800394 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.833818 mds.0 10.252.24.25:6802/3144 25 : [DBG] reconnect by client.11042 10.252.24.25:0/2614196857 after 0.000622 -21631> 2013-04-03 10:34:26.800413 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.835300 mds.0 10.252.24.25:6802/3144 26 : [DBG] reconnect by client.11041 10.252.24.25:0/3745741276 after 0.002096 -21630> 2013-04-03 10:34:26.800432 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.835370 mds.0 10.252.24.25:6802/3144 27 : [DBG] reconnect by client.11074 10.252.24.32:0/3871608469 after 0.002174 -21629> 2013-04-03 10:34:26.800449 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.835418 mds.0 10.252.24.25:6802/3144 28 : [DBG] reconnect by client.11054 10.252.24.30:0/2446411061 after 0.002224 -21628> 2013-04-03 10:34:26.800467 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.835463 mds.0 10.252.24.25:6802/3144 29 : [DBG] reconnect by client.11045 10.252.24.25:0/2151250553 after 0.002268 -21627> 2013-04-03 10:34:26.800485 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.835663 mds.0 10.252.24.25:6802/3144 30 : [DBG] reconnect by client.11802 10.252.24.30:0/517 after 0.002465 -21626> 2013-04-03 10:34:26.800510 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.835708 mds.0 10.252.24.25:6802/3144 31 : [DBG] reconnect by client.11071 10.252.24.32:0/1396772501 after 0.002514 -21625> 2013-04-03 10:34:26.800531 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.835801 mds.0 10.252.24.25:6802/3144 32 : [DBG] reconnect by client.11051 10.252.24.30:0/802047285 after 0.002603 -21624> 2013-04-03 10:34:26.800549 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:16.859400 mds.0 10.252.24.25:6802/3144 33 : [DBG] reconnect by client.11228 10.252.24.25:0/1941 after 0.026202 -21623> 2013-04-03 10:34:26.800567 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 update_from_paxos applying incremental log 393979 2013-04-03 10:34:25.761508 mon.0 10.252.24.25:6789/0 31 : [INF] pgmap v28973: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 177B/s rd, 64792B/s wr, 27op/s -21622> 2013-04-03 10:34:26.800607 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 check_subs -21621> 2013-04-03 10:34:26.800636 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 create_pending v 393980 -21620> 2013-04-03 10:34:26.800657 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 _updated_log for mon.0 10.252.24.25:6789/0 -21619> 2013-04-03 10:34:26.800670 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 31) v1 -- ?+0 0x2294700 con 0x1bac160 -21618> 2013-04-03 10:34:26.800677 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393979 _updated_log for mds.0 10.252.24.25:6802/3144 -21617> 2013-04-03 10:34:26.800690 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(log(last 33) v1 tid 17407) v2 -- ?+0 0x20fc240 con 0x208a840 -21616> 2013-04-03 10:34:26.800721 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427226) propose_queued 427227 654 bytes -21615> 2013-04-03 10:34:26.800946 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427226 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2808000 -21614> 2013-04-03 10:34:26.800986 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 31) v1 ==== 0+0+0 (0 0 0) 0x2294700 con 0x1bac160 -21613> 2013-04-03 10:34:26.800994 7fcdffcdb700 10 handle_log_ack log(last 31) v1 -21612> 2013-04-03 10:34:26.800997 7fcdffcdb700 10 logged 2013-04-03 10:34:25.761508 mon.0 10.252.24.25:6789/0 31 : [INF] pgmap v28973: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 177B/s rd, 64792B/s wr, 27op/s -21611> 2013-04-03 10:34:26.824300 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112970 ==== paxos(lease_ack lc 427226 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (868140059 0 0) 0x2808000 con 0x208a840 -21610> 2013-04-03 10:34:26.837877 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112971 ==== paxos(accept lc 427226 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1579495306 0 0) 0x26dca00 con 0x208a840 -21609> 2013-04-03 10:34:26.838154 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427227 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2808000 -21608> 2013-04-03 10:34:26.838171 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427227 fc 427192 pn 0 opn 0) v3 -- ?+0 0x26dc780 -21607> 2013-04-03 10:34:26.838237 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28973 update_from_paxos applying incremental 28974 -21606> 2013-04-03 10:34:26.838257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 v28974: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 37124B/s wr, 16op/s -21605> 2013-04-03 10:34:26.838276 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 send_pg_creates to 0 pgs -21604> 2013-04-03 10:34:26.838280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 update_logger -21603> 2013-04-03 10:34:26.838290 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 create_pending v 28975 -21602> 2013-04-03 10:34:26.838299 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28974 _updated_stats for osd.3 10.252.24.88:6801/17538 -21601> 2013-04-03 10:34:26.838305 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30434) v1 tid 17406) v2 -- ?+0 0x20adb40 con 0x208a840 -21600> 2013-04-03 10:34:26.838315 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 check_osd_map already seen 599 >= 599 -21599> 2013-04-03 10:34:26.838317 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 update_logger -21598> 2013-04-03 10:34:26.838327 7fcdffcdb700 0 log [INF] : pgmap v28974: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 37124B/s wr, 16op/s -21597> 2013-04-03 10:34:26.838329 7fcdffcdb700 10 send_log to self -21596> 2013-04-03 10:34:26.838330 7fcdffcdb700 10 log_queue is 1 last_log 32 sent 31 num 1 unsent 1 sending 1 -21595> 2013-04-03 10:34:26.838331 7fcdffcdb700 10 will send 2013-04-03 10:34:26.838328 mon.0 10.252.24.25:6789/0 32 : [INF] pgmap v28974: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 37124B/s wr, 16op/s -21594> 2013-04-03 10:34:26.838337 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2055000 -21593> 2013-04-03 10:34:26.838381 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos -21592> 2013-04-03 10:34:26.838397 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos version 393979 summary v 393979 -21591> 2013-04-03 10:34:26.838415 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21590> 2013-04-03 10:34:26.838435 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2055000 con 0x1bac160 -21589> 2013-04-03 10:34:26.838456 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos -21588> 2013-04-03 10:34:26.838470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos version 393979 summary v 393979 -21587> 2013-04-03 10:34:26.838482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21586> 2013-04-03 10:34:26.838494 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 preprocess_log log(1 entries) v1 from mon.0 -21585> 2013-04-03 10:34:26.838509 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21584> 2013-04-03 10:34:26.838521 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 prepare_log log(1 entries) v1 from mon.0 -21583> 2013-04-03 10:34:26.838532 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 logging 2013-04-03 10:34:26.838328 mon.0 10.252.24.25:6789/0 32 : [INF] pgmap v28974: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 37124B/s wr, 16op/s -21582> 2013-04-03 10:34:26.862921 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112972 ==== paxos(lease_ack lc 427227 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (495677043 0 0) 0x26dc780 con 0x208a840 -21581> 2013-04-03 10:34:26.862957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos -21580> 2013-04-03 10:34:26.862972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393979 update_from_paxos version 393979 summary v 393979 -21579> 2013-04-03 10:34:26.862986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21578> 2013-04-03 10:34:27.576356 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 12 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140300 v187) v2 ==== 289+0+0 (3351126167 0 0) 0x2145b00 con 0x2118420 -21577> 2013-04-03 10:34:27.576407 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140300 v187) v2 from mds.? 10.252.58.31:6800/10908 -21576> 2013-04-03 10:34:27.576418 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140300 v187) v2 -- ?+0 0x2145dc0 con 0x2118420 -21575> 2013-04-03 10:34:27.838032 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393979 encode_pending v393980 -21574> 2013-04-03 10:34:27.838059 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427227) queue_proposal bl 445 bytes; ctx = 0x1b91850 -21573> 2013-04-03 10:34:27.838067 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427227) propose_queued 427228 445 bytes -21572> 2013-04-03 10:34:27.839202 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427227 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc500 -21571> 2013-04-03 10:34:27.865984 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112973 ==== paxos(accept lc 427227 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (485498519 0 0) 0x26dc500 con 0x208a840 -21570> 2013-04-03 10:34:27.870630 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427228 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc780 -21569> 2013-04-03 10:34:27.870665 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427228 fc 427192 pn 0 opn 0) v3 -- ?+0 0x26dca00 -21568> 2013-04-03 10:34:27.870706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos -21567> 2013-04-03 10:34:27.870749 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos version 393980 summary v 393979 -21566> 2013-04-03 10:34:27.870763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos latest full 393798 -21565> 2013-04-03 10:34:27.870782 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393980 update_from_paxos applying incremental log 393980 2013-04-03 10:34:26.838328 mon.0 10.252.24.25:6789/0 32 : [INF] pgmap v28974: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 37124B/s wr, 16op/s -21564> 2013-04-03 10:34:27.870829 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 check_subs -21563> 2013-04-03 10:34:27.870858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 create_pending v 393981 -21562> 2013-04-03 10:34:27.870879 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393980 _updated_log for mon.0 10.252.24.25:6789/0 -21561> 2013-04-03 10:34:27.870892 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 32) v1 -- ?+0 0x2294700 con 0x1bac160 -21560> 2013-04-03 10:34:27.870939 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos -21559> 2013-04-03 10:34:27.870954 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos version 393980 summary v 393980 -21558> 2013-04-03 10:34:27.870972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21557> 2013-04-03 10:34:27.870992 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 32) v1 ==== 0+0+0 (0 0 0) 0x2294700 con 0x1bac160 -21556> 2013-04-03 10:34:27.871000 7fcdffcdb700 10 handle_log_ack log(last 32) v1 -21555> 2013-04-03 10:34:27.871001 7fcdffcdb700 10 logged 2013-04-03 10:34:26.838328 mon.0 10.252.24.25:6789/0 32 : [INF] pgmap v28974: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 37124B/s wr, 16op/s -21554> 2013-04-03 10:34:27.887987 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112974 ==== paxos(lease_ack lc 427228 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (3996475307 0 0) 0x26dca00 con 0x208a840 -21553> 2013-04-03 10:34:27.888032 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos -21552> 2013-04-03 10:34:27.888048 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos version 393980 summary v 393980 -21551> 2013-04-03 10:34:27.888062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21550> 2013-04-03 10:34:28.322920 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28974 check_down_pgs -21549> 2013-04-03 10:34:28.322956 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28974 v28974: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 37124B/s wr, 16op/s -21548> 2013-04-03 10:34:28.322982 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -21547> 2013-04-03 10:34:28.323005 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in -21546> 2013-04-03 10:34:28.323039 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 -21545> 2013-04-03 10:34:28.323068 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos -21544> 2013-04-03 10:34:28.323085 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos version 393980 summary v 393980 -21543> 2013-04-03 10:34:28.323098 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393980 log -21542> 2013-04-03 10:34:28.323114 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21541> 2013-04-03 10:34:28.323132 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -21540> 2013-04-03 10:34:28.922171 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112975 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 15 v187) v2) to leader v1 ==== 560+0+0 (980452016 0 0) 0x2146b80 con 0x208a840 -21539> 2013-04-03 10:34:28.922193 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -21538> 2013-04-03 10:34:28.922202 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2145dc0 from 10.252.24.88:6789/0 -21537> 2013-04-03 10:34:28.922253 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 15 v187) v2 from mds.0 10.252.24.25:6802/3144 -21536> 2013-04-03 10:34:28.922264 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 15 v187) v2 tid 17408) v2 -- ?+0 0x20ad900 con 0x208a840 -21535> 2013-04-03 10:34:29.376557 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 12 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21671 v187) v2 ==== 289+0+0 (73128000 0 0) 0x1baeb80 con 0x200d420 -21534> 2013-04-03 10:34:29.376628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21671 v187) v2 from mds.? 10.252.24.88:6800/17213 -21533> 2013-04-03 10:34:29.376640 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21671 v187) v2 -- ?+0 0x2146b80 con 0x200d420 -21532> 2013-04-03 10:34:29.826489 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112976 ==== forward(pg_stats(1 pgs tid 526 v 599) v1) to leader v1 ==== 753+0+0 (3307766828 0 0) 0x2145b00 con 0x208a840 -21531> 2013-04-03 10:34:29.826516 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 -21530> 2013-04-03 10:34:29.826526 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad900 from 10.252.24.88:6789/0 -21529> 2013-04-03 10:34:29.826574 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 preprocess_query pg_stats(1 pgs tid 526 v 599) v1 from osd.2 10.252.24.30:6801/10373 -21528> 2013-04-03 10:34:29.826596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 prepare_update pg_stats(1 pgs tid 526 v 599) v1 from osd.2 10.252.24.30:6801/10373 -21527> 2013-04-03 10:34:29.826602 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 prepare_pg_stats pg_stats(1 pgs tid 526 v 599) v1 from osd.2 -21526> 2013-04-03 10:34:29.826608 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28974 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[])) -21525> 2013-04-03 10:34:29.876789 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28974 encode_pending v 28975 -21524> 2013-04-03 10:34:29.876826 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427228) queue_proposal bl 650 bytes; ctx = 0x1b91810 -21523> 2013-04-03 10:34:29.876835 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427228) propose_queued 427229 650 bytes -21522> 2013-04-03 10:34:29.877818 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427228 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc280 -21521> 2013-04-03 10:34:29.912125 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112977 ==== paxos(accept lc 427228 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2326913021 0 0) 0x26dc280 con 0x208a840 -21520> 2013-04-03 10:34:29.913004 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427229 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dca00 -21519> 2013-04-03 10:34:29.913037 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427229 fc 427192 pn 0 opn 0) v3 -- ?+0 0x26dc500 -21518> 2013-04-03 10:34:29.913106 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28974 update_from_paxos applying incremental 28975 -21517> 2013-04-03 10:34:29.913121 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 v28975: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2157B/s wr, 0op/s -21516> 2013-04-03 10:34:29.913140 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 send_pg_creates to 0 pgs -21515> 2013-04-03 10:34:29.913144 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 update_logger -21514> 2013-04-03 10:34:29.913153 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 create_pending v 28976 -21513> 2013-04-03 10:34:29.913162 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28975 _updated_stats for osd.2 10.252.24.30:6801/10373 -21512> 2013-04-03 10:34:29.913171 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 526) v1 tid 17409) v2 -- ?+0 0x20adb40 con 0x208a840 -21511> 2013-04-03 10:34:29.913182 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 check_osd_map already seen 599 >= 599 -21510> 2013-04-03 10:34:29.913185 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 update_logger -21509> 2013-04-03 10:34:29.913197 7fcdffcdb700 0 log [INF] : pgmap v28975: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2157B/s wr, 0op/s -21508> 2013-04-03 10:34:29.913199 7fcdffcdb700 10 send_log to self -21507> 2013-04-03 10:34:29.913199 7fcdffcdb700 10 log_queue is 1 last_log 33 sent 32 num 1 unsent 1 sending 1 -21506> 2013-04-03 10:34:29.913201 7fcdffcdb700 10 will send 2013-04-03 10:34:29.913198 mon.0 10.252.24.25:6789/0 33 : [INF] pgmap v28975: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2157B/s wr, 0op/s -21505> 2013-04-03 10:34:29.913209 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2055000 -21504> 2013-04-03 10:34:29.913255 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos -21503> 2013-04-03 10:34:29.913273 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos version 393980 summary v 393980 -21502> 2013-04-03 10:34:29.913291 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21501> 2013-04-03 10:34:29.913319 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2055000 con 0x1bac160 -21500> 2013-04-03 10:34:29.913342 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos -21499> 2013-04-03 10:34:29.913356 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos version 393980 summary v 393980 -21498> 2013-04-03 10:34:29.913365 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21497> 2013-04-03 10:34:29.913375 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 preprocess_log log(1 entries) v1 from mon.0 -21496> 2013-04-03 10:34:29.913391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21495> 2013-04-03 10:34:29.913400 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 prepare_log log(1 entries) v1 from mon.0 -21494> 2013-04-03 10:34:29.913409 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 logging 2013-04-03 10:34:29.913198 mon.0 10.252.24.25:6789/0 33 : [INF] pgmap v28975: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2157B/s wr, 0op/s -21493> 2013-04-03 10:34:29.938193 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112978 ==== paxos(lease_ack lc 427229 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (2381932422 0 0) 0x26dc500 con 0x208a840 -21492> 2013-04-03 10:34:29.938232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos -21491> 2013-04-03 10:34:29.938248 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393980 update_from_paxos version 393980 summary v 393980 -21490> 2013-04-03 10:34:29.938262 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21489> 2013-04-03 10:34:30.554716 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112979 ==== forward(pg_stats(1 pgs tid 7 v 599) v1) to leader v1 ==== 757+0+0 (3089936805 0 0) 0x2145840 con 0x208a840 -21488> 2013-04-03 10:34:30.554739 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -21487> 2013-04-03 10:34:30.554749 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -21486> 2013-04-03 10:34:30.554800 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 preprocess_query pg_stats(1 pgs tid 7 v 599) v1 from osd.1 10.252.24.25:6805/3572 -21485> 2013-04-03 10:34:30.554822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 prepare_update pg_stats(1 pgs tid 7 v 599) v1 from osd.1 10.252.24.25:6805/3572 -21484> 2013-04-03 10:34:30.554828 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 prepare_pg_stats pg_stats(1 pgs tid 7 v 599) v1 from osd.1 -21483> 2013-04-03 10:34:30.554834 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28975 got osd.1 osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -21482> 2013-04-03 10:34:30.912328 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393980 encode_pending v393981 -21481> 2013-04-03 10:34:30.912355 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427229) queue_proposal bl 443 bytes; ctx = 0x1b91870 -21480> 2013-04-03 10:34:30.912362 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427229) propose_queued 427230 443 bytes -21479> 2013-04-03 10:34:30.913450 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427229 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc000 -21478> 2013-04-03 10:34:30.913543 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28975 encode_pending v 28976 -21477> 2013-04-03 10:34:30.913581 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427192..427229) queue_proposal bl 654 bytes; ctx = 0x1b918f0 -21476> 2013-04-03 10:34:30.913589 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427192..427229) propose_new_value not active; proposal queued -21475> 2013-04-03 10:34:30.963266 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112980 ==== paxos(accept lc 427229 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4089746254 0 0) 0x26dc000 con 0x208a840 -21474> 2013-04-03 10:34:30.963563 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427230 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc500 -21473> 2013-04-03 10:34:30.963594 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427230 fc 427192 pn 0 opn 0) v3 -- ?+0 0x26dc280 -21472> 2013-04-03 10:34:30.963635 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos -21471> 2013-04-03 10:34:30.963660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos version 393981 summary v 393980 -21470> 2013-04-03 10:34:30.963679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos latest full 393798 -21469> 2013-04-03 10:34:30.963698 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393981 update_from_paxos applying incremental log 393981 2013-04-03 10:34:29.913198 mon.0 10.252.24.25:6789/0 33 : [INF] pgmap v28975: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2157B/s wr, 0op/s -21468> 2013-04-03 10:34:30.963750 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 check_subs -21467> 2013-04-03 10:34:30.963781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 create_pending v 393982 -21466> 2013-04-03 10:34:30.963802 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393981 _updated_log for mon.0 10.252.24.25:6789/0 -21465> 2013-04-03 10:34:30.963815 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 33) v1 -- ?+0 0x22241c0 con 0x1bac160 -21464> 2013-04-03 10:34:30.963840 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427230) propose_queued 427231 654 bytes -21463> 2013-04-03 10:34:30.964644 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427230 fc 0 pn 17600 opn 0) v3 -- ?+0 0x282f280 -21462> 2013-04-03 10:34:30.964678 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 33) v1 ==== 0+0+0 (0 0 0) 0x22241c0 con 0x1bac160 -21461> 2013-04-03 10:34:30.964697 7fcdffcdb700 10 handle_log_ack log(last 33) v1 -21460> 2013-04-03 10:34:30.964700 7fcdffcdb700 10 logged 2013-04-03 10:34:29.913198 mon.0 10.252.24.25:6789/0 33 : [INF] pgmap v28975: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2157B/s wr, 0op/s -21459> 2013-04-03 10:34:30.975603 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112981 ==== paxos(lease_ack lc 427230 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (3082886376 0 0) 0x282f280 con 0x208a840 -21458> 2013-04-03 10:34:30.990874 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112982 ==== paxos(accept lc 427230 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2014771319 0 0) 0x26dc280 con 0x208a840 -21457> 2013-04-03 10:34:30.991220 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427231 fc 0 pn 17600 opn 0) v3 -- ?+0 0x282f280 -21456> 2013-04-03 10:34:30.991238 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427231 fc 427192 pn 0 opn 0) v3 -- ?+0 0x26dc000 -21455> 2013-04-03 10:34:30.991321 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28975 update_from_paxos applying incremental 28976 -21454> 2013-04-03 10:34:30.991342 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 v28976: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2220B/s wr, 0op/s -21453> 2013-04-03 10:34:30.991362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 send_pg_creates to 0 pgs -21452> 2013-04-03 10:34:30.991366 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 update_logger -21451> 2013-04-03 10:34:30.991376 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 create_pending v 28977 -21450> 2013-04-03 10:34:30.991385 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28976 _updated_stats for osd.1 10.252.24.25:6805/3572 -21449> 2013-04-03 10:34:30.991392 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 7) v1 tid 17410) v2 -- ?+0 0x20ad900 con 0x208a840 -21448> 2013-04-03 10:34:30.991407 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 check_osd_map already seen 599 >= 599 -21447> 2013-04-03 10:34:30.991410 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 update_logger -21446> 2013-04-03 10:34:30.991422 7fcdffcdb700 0 log [INF] : pgmap v28976: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2220B/s wr, 0op/s -21445> 2013-04-03 10:34:30.991424 7fcdffcdb700 10 send_log to self -21444> 2013-04-03 10:34:30.991424 7fcdffcdb700 10 log_queue is 1 last_log 34 sent 33 num 1 unsent 1 sending 1 -21443> 2013-04-03 10:34:30.991426 7fcdffcdb700 10 will send 2013-04-03 10:34:30.991423 mon.0 10.252.24.25:6789/0 34 : [INF] pgmap v28976: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2220B/s wr, 0op/s -21442> 2013-04-03 10:34:30.991432 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x1c74400 -21441> 2013-04-03 10:34:30.991478 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos -21440> 2013-04-03 10:34:30.991496 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos version 393981 summary v 393981 -21439> 2013-04-03 10:34:30.991514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21438> 2013-04-03 10:34:30.991534 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x1c74400 con 0x1bac160 -21437> 2013-04-03 10:34:30.991555 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos -21436> 2013-04-03 10:34:30.991570 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos version 393981 summary v 393981 -21435> 2013-04-03 10:34:30.991578 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21434> 2013-04-03 10:34:30.991590 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 preprocess_log log(1 entries) v1 from mon.0 -21433> 2013-04-03 10:34:30.991607 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21432> 2013-04-03 10:34:30.991619 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 prepare_log log(1 entries) v1 from mon.0 -21431> 2013-04-03 10:34:30.991630 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 logging 2013-04-03 10:34:30.991423 mon.0 10.252.24.25:6789/0 34 : [INF] pgmap v28976: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2220B/s wr, 0op/s -21430> 2013-04-03 10:34:31.002890 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112983 ==== paxos(lease_ack lc 427231 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (558147781 0 0) 0x26dc000 con 0x208a840 -21429> 2013-04-03 10:34:31.002943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos -21428> 2013-04-03 10:34:31.002961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393981 update_from_paxos version 393981 summary v 393981 -21427> 2013-04-03 10:34:31.002976 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21426> 2013-04-03 10:34:31.576503 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 13 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140301 v187) v2 ==== 289+0+0 (3838877674 0 0) 0x20e3840 con 0x2118420 -21425> 2013-04-03 10:34:31.576551 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140301 v187) v2 from mds.? 10.252.58.31:6800/10908 -21424> 2013-04-03 10:34:31.576562 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140301 v187) v2 -- ?+0 0x2145840 con 0x2118420 -21423> 2013-04-03 10:34:31.991089 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393981 encode_pending v393982 -21422> 2013-04-03 10:34:31.991117 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427231) queue_proposal bl 443 bytes; ctx = 0x1b91530 -21421> 2013-04-03 10:34:31.991125 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427231) propose_queued 427232 443 bytes -21420> 2013-04-03 10:34:31.992367 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427231 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2715c80 -21419> 2013-04-03 10:34:32.018776 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112984 ==== paxos(accept lc 427231 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2050367865 0 0) 0x2715c80 con 0x208a840 -21418> 2013-04-03 10:34:32.019543 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427232 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc000 -21417> 2013-04-03 10:34:32.019561 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427232 fc 427192 pn 0 opn 0) v3 -- ?+0 0x26dc280 -21416> 2013-04-03 10:34:32.019598 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos -21415> 2013-04-03 10:34:32.019625 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos version 393982 summary v 393981 -21414> 2013-04-03 10:34:32.019639 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos latest full 393798 -21413> 2013-04-03 10:34:32.019657 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393982 update_from_paxos applying incremental log 393982 2013-04-03 10:34:30.991423 mon.0 10.252.24.25:6789/0 34 : [INF] pgmap v28976: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2220B/s wr, 0op/s -21412> 2013-04-03 10:34:32.019698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 check_subs -21411> 2013-04-03 10:34:32.019727 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 create_pending v 393983 -21410> 2013-04-03 10:34:32.019749 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393982 _updated_log for mon.0 10.252.24.25:6789/0 -21409> 2013-04-03 10:34:32.019761 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 34) v1 -- ?+0 0x22241c0 con 0x1bac160 -21408> 2013-04-03 10:34:32.020012 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos -21407> 2013-04-03 10:34:32.020031 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos version 393982 summary v 393982 -21406> 2013-04-03 10:34:32.020051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21405> 2013-04-03 10:34:32.020071 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 34) v1 ==== 0+0+0 (0 0 0) 0x22241c0 con 0x1bac160 -21404> 2013-04-03 10:34:32.020081 7fcdffcdb700 10 handle_log_ack log(last 34) v1 -21403> 2013-04-03 10:34:32.020083 7fcdffcdb700 10 logged 2013-04-03 10:34:30.991423 mon.0 10.252.24.25:6789/0 34 : [INF] pgmap v28976: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2220B/s wr, 0op/s -21402> 2013-04-03 10:34:32.043976 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112985 ==== paxos(lease_ack lc 427232 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (387314122 0 0) 0x26dc280 con 0x208a840 -21401> 2013-04-03 10:34:32.044022 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos -21400> 2013-04-03 10:34:32.044039 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos version 393982 summary v 393982 -21399> 2013-04-03 10:34:32.044054 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21398> 2013-04-03 10:34:32.922325 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112986 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 16 v187) v2) to leader v1 ==== 560+0+0 (1382823486 0 0) 0x20e4340 con 0x208a840 -21397> 2013-04-03 10:34:32.922346 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -21396> 2013-04-03 10:34:32.922355 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e3b00 from 10.252.24.88:6789/0 -21395> 2013-04-03 10:34:32.922406 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 16 v187) v2 from mds.0 10.252.24.25:6802/3144 -21394> 2013-04-03 10:34:32.922418 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 16 v187) v2 tid 17411) v2 -- ?+0 0x20adb40 con 0x208a840 -21393> 2013-04-03 10:34:33.323254 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28976 check_down_pgs -21392> 2013-04-03 10:34:33.323292 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28976 v28976: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail; 2220B/s wr, 0op/s -21391> 2013-04-03 10:34:33.323318 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -21390> 2013-04-03 10:34:33.323340 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 e599: 5 osds: 5 up, 5 in -21389> 2013-04-03 10:34:33.323376 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e599 min_last_epoch_clean 593 -21388> 2013-04-03 10:34:33.323404 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos -21387> 2013-04-03 10:34:33.323421 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos version 393982 summary v 393982 -21386> 2013-04-03 10:34:33.323430 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393982 log -21385> 2013-04-03 10:34:33.323446 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21384> 2013-04-03 10:34:33.323464 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -21383> 2013-04-03 10:34:33.376653 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 13 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21672 v187) v2 ==== 289+0+0 (4073417002 0 0) 0x1bae8c0 con 0x200d420 -21382> 2013-04-03 10:34:33.376701 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21672 v187) v2 from mds.? 10.252.24.88:6800/17213 -21381> 2013-04-03 10:34:33.376711 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21672 v187) v2 -- ?+0 0x20e4340 con 0x200d420 -21380> 2013-04-03 10:34:34.826870 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112987 ==== forward(pg_stats(1 pgs tid 527 v 599) v1) to leader v1 ==== 753+0+0 (2463485169 0 0) 0x20e3840 con 0x208a840 -21379> 2013-04-03 10:34:34.826890 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/10373 via mon.2 10.252.24.88:6789/0 -21378> 2013-04-03 10:34:34.826900 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -21377> 2013-04-03 10:34:34.826952 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 preprocess_query pg_stats(1 pgs tid 527 v 599) v1 from osd.2 10.252.24.30:6801/10373 -21376> 2013-04-03 10:34:34.826974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 prepare_update pg_stats(1 pgs tid 527 v 599) v1 from osd.2 10.252.24.30:6801/10373 -21375> 2013-04-03 10:34:34.826980 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 prepare_pg_stats pg_stats(1 pgs tid 527 v 599) v1 from osd.2 -21374> 2013-04-03 10:34:34.826986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28976 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[])) -21373> 2013-04-03 10:34:34.877161 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28976 encode_pending v 28977 -21372> 2013-04-03 10:34:34.877205 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427232) queue_proposal bl 650 bytes; ctx = 0x1b917f0 -21371> 2013-04-03 10:34:34.877218 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427232) propose_queued 427233 650 bytes -21370> 2013-04-03 10:34:34.878376 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427232 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2715a00 -21369> 2013-04-03 10:34:34.907420 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112988 ==== paxos(accept lc 427232 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (842572431 0 0) 0x2715a00 con 0x208a840 -21368> 2013-04-03 10:34:34.907842 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427233 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc280 -21367> 2013-04-03 10:34:34.907873 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427233 fc 427192 pn 0 opn 0) v3 -- ?+0 0x2715c80 -21366> 2013-04-03 10:34:34.907965 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28976 update_from_paxos applying incremental 28977 -21365> 2013-04-03 10:34:34.907984 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 v28977: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s -21364> 2013-04-03 10:34:34.908002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 send_pg_creates to 0 pgs -21363> 2013-04-03 10:34:34.908004 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 update_logger -21362> 2013-04-03 10:34:34.908014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 create_pending v 28978 -21361> 2013-04-03 10:34:34.908023 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28977 _updated_stats for osd.2 10.252.24.30:6801/10373 -21360> 2013-04-03 10:34:34.908030 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 527) v1 tid 17412) v2 -- ?+0 0x20ad900 con 0x208a840 -21359> 2013-04-03 10:34:34.908042 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 check_osd_map already seen 599 >= 599 -21358> 2013-04-03 10:34:34.908046 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 update_logger -21357> 2013-04-03 10:34:34.908057 7fcdffcdb700 0 log [INF] : pgmap v28977: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s -21356> 2013-04-03 10:34:34.908059 7fcdffcdb700 10 send_log to self -21355> 2013-04-03 10:34:34.908059 7fcdffcdb700 10 log_queue is 1 last_log 35 sent 34 num 1 unsent 1 sending 1 -21354> 2013-04-03 10:34:34.908067 7fcdffcdb700 10 will send 2013-04-03 10:34:34.908058 mon.0 10.252.24.25:6789/0 35 : [INF] pgmap v28977: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s -21353> 2013-04-03 10:34:34.908075 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x1c74400 -21352> 2013-04-03 10:34:34.908145 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427192..427233) queue_proposal bl 987 bytes; ctx = 0x1b918d0 -21351> 2013-04-03 10:34:34.908154 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427192..427233) propose_queued 427234 987 bytes -21350> 2013-04-03 10:34:34.908390 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427233 fc 0 pn 17600 opn 0) v3 -- ?+0 0x282f000 -21349> 2013-04-03 10:34:34.908426 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x1c74400 con 0x1bac160 -21348> 2013-04-03 10:34:34.908468 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos -21347> 2013-04-03 10:34:34.908484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos version 393982 summary v 393982 -21346> 2013-04-03 10:34:34.908493 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21345> 2013-04-03 10:34:34.908504 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 preprocess_log log(1 entries) v1 from mon.0 -21344> 2013-04-03 10:34:34.908520 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21343> 2013-04-03 10:34:34.908531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 prepare_log log(1 entries) v1 from mon.0 -21342> 2013-04-03 10:34:34.908540 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 logging 2013-04-03 10:34:34.908058 mon.0 10.252.24.25:6789/0 35 : [INF] pgmap v28977: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s -21341> 2013-04-03 10:34:34.920579 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112989 ==== paxos(lease_ack lc 427233 fc 427192 pn 0 opn 0) v3 ==== 80+0+0 (519820236 0 0) 0x282f000 con 0x208a840 -21340> 2013-04-03 10:34:34.945772 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112990 ==== paxos(accept lc 427233 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4246516862 0 0) 0x2715c80 con 0x208a840 -21339> 2013-04-03 10:34:34.946109 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427234 fc 0 pn 17600 opn 0) v3 -- ?+0 0x282f000 -21338> 2013-04-03 10:34:34.946128 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427234 fc 427192 pn 0 opn 0) v3 -- ?+0 0x2715a00 -21337> 2013-04-03 10:34:34.946197 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos -21336> 2013-04-03 10:34:34.946213 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos version 393982 summary v 393982 -21335> 2013-04-03 10:34:34.946233 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21334> 2013-04-03 10:34:34.957674 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112991 ==== paxos(lease_ack lc 427234 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (3958296740 0 0) 0x2715a00 con 0x208a840 -21333> 2013-04-03 10:34:34.957706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos -21332> 2013-04-03 10:34:34.957722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393982 update_from_paxos version 393982 summary v 393982 -21331> 2013-04-03 10:34:34.957736 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21330> 2013-04-03 10:34:35.555250 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112992 ==== forward(pg_stats(1 pgs tid 8 v 599) v1) to leader v1 ==== 757+0+0 (3079325074 0 0) 0x2146b80 con 0x208a840 -21329> 2013-04-03 10:34:35.555262 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -21328> 2013-04-03 10:34:35.555270 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20ad900 from 10.252.24.88:6789/0 -21327> 2013-04-03 10:34:35.555506 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 preprocess_query pg_stats(1 pgs tid 8 v 599) v1 from osd.1 10.252.24.25:6805/3572 -21326> 2013-04-03 10:34:35.555526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 prepare_update pg_stats(1 pgs tid 8 v 599) v1 from osd.1 10.252.24.25:6805/3572 -21325> 2013-04-03 10:34:35.555532 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 prepare_pg_stats pg_stats(1 pgs tid 8 v 599) v1 from osd.1 -21324> 2013-04-03 10:34:35.555537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28977 got osd.1 osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -21323> 2013-04-03 10:34:35.576593 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 14 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140302 v187) v2 ==== 289+0+0 (2170773101 0 0) 0x20e3dc0 con 0x2118420 -21322> 2013-04-03 10:34:35.576646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140302 v187) v2 from mds.? 10.252.58.31:6800/10908 -21321> 2013-04-03 10:34:35.576658 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140302 v187) v2 -- ?+0 0x2146b80 con 0x2118420 -21320> 2013-04-03 10:34:35.907636 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393982 encode_pending v393983 -21319> 2013-04-03 10:34:35.907673 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427223..427234) queue_proposal bl 443 bytes; ctx = 0x1b91820 -21318> 2013-04-03 10:34:35.907681 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427223..427234) propose_queued 427235 443 bytes -21317> 2013-04-03 10:34:35.908745 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427234 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2715780 -21316> 2013-04-03 10:34:35.931972 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112993 ==== paxos(accept lc 427234 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (100588788 0 0) 0x2715780 con 0x208a840 -21315> 2013-04-03 10:34:35.932265 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427235 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2715a00 -21314> 2013-04-03 10:34:35.932296 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427235 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2715c80 -21313> 2013-04-03 10:34:35.932340 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos -21312> 2013-04-03 10:34:35.932366 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos version 393983 summary v 393982 -21311> 2013-04-03 10:34:35.932385 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos latest full 393798 -21310> 2013-04-03 10:34:35.932403 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393983 update_from_paxos applying incremental log 393983 2013-04-03 10:34:34.908058 mon.0 10.252.24.25:6789/0 35 : [INF] pgmap v28977: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s -21309> 2013-04-03 10:34:35.932450 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 check_subs -21308> 2013-04-03 10:34:35.932480 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 create_pending v 393984 -21307> 2013-04-03 10:34:35.932501 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393983 _updated_log for mon.0 10.252.24.25:6789/0 -21306> 2013-04-03 10:34:35.932513 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 35) v1 -- ?+0 0x2224e00 con 0x1bac160 -21305> 2013-04-03 10:34:35.932562 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos -21304> 2013-04-03 10:34:35.932577 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos version 393983 summary v 393983 -21303> 2013-04-03 10:34:35.932594 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21302> 2013-04-03 10:34:35.932614 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 35) v1 ==== 0+0+0 (0 0 0) 0x2224e00 con 0x1bac160 -21301> 2013-04-03 10:34:35.932622 7fcdffcdb700 10 handle_log_ack log(last 35) v1 -21300> 2013-04-03 10:34:35.932625 7fcdffcdb700 10 logged 2013-04-03 10:34:34.908058 mon.0 10.252.24.25:6789/0 35 : [INF] pgmap v28977: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s -21299> 2013-04-03 10:34:35.943841 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112994 ==== paxos(lease_ack lc 427235 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (7316611 0 0) 0x2715c80 con 0x208a840 -21298> 2013-04-03 10:34:35.943876 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos -21297> 2013-04-03 10:34:35.943892 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos version 393983 summary v 393983 -21296> 2013-04-03 10:34:35.943907 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21295> 2013-04-03 10:34:35.945928 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28977 encode_pending v 28978 -21294> 2013-04-03 10:34:35.945961 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427223..427235) queue_proposal bl 654 bytes; ctx = 0x1b91510 -21293> 2013-04-03 10:34:35.945971 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427223..427235) propose_queued 427236 654 bytes -21292> 2013-04-03 10:34:35.947245 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427235 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2715500 -21291> 2013-04-03 10:34:35.977358 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112995 ==== paxos(accept lc 427235 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3309182816 0 0) 0x2715500 con 0x208a840 -21290> 2013-04-03 10:34:35.977679 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427236 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2715c80 -21289> 2013-04-03 10:34:35.977711 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427236 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2715780 -21288> 2013-04-03 10:34:35.977781 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28977 update_from_paxos applying incremental 28978 -21287> 2013-04-03 10:34:35.977800 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28978 v28978: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 7732B/s wr, 2op/s -21286> 2013-04-03 10:34:35.977827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28978 send_pg_creates to 0 pgs -21285> 2013-04-03 10:34:35.977831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28978 update_logger -21284> 2013-04-03 10:34:35.977841 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28978 create_pending v 28979 -21283> 2013-04-03 10:34:35.977849 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28978 _updated_stats for osd.1 10.252.24.25:6805/3572 -21282> 2013-04-03 10:34:35.977856 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 8) v1 tid 17413) v2 -- ?+0 0x20adb40 con 0x208a840 -21281> 2013-04-03 10:34:35.977865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28978 check_osd_map already seen 599 >= 599 -21280> 2013-04-03 10:34:35.977868 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28978 update_logger -21279> 2013-04-03 10:34:35.977880 7fcdffcdb700 0 log [INF] : pgmap v28978: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 7732B/s wr, 2op/s -21278> 2013-04-03 10:34:35.977881 7fcdffcdb700 10 send_log to self -21277> 2013-04-03 10:34:35.977882 7fcdffcdb700 10 log_queue is 1 last_log 36 sent 35 num 1 unsent 1 sending 1 -21276> 2013-04-03 10:34:35.977884 7fcdffcdb700 10 will send 2013-04-03 10:34:35.977881 mon.0 10.252.24.25:6789/0 36 : [INF] pgmap v28978: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 7732B/s wr, 2op/s -21275> 2013-04-03 10:34:35.977890 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2169000 -21274> 2013-04-03 10:34:35.977932 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos -21273> 2013-04-03 10:34:35.977948 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos version 393983 summary v 393983 -21272> 2013-04-03 10:34:35.977964 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21271> 2013-04-03 10:34:35.977982 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2169000 con 0x1bac160 -21270> 2013-04-03 10:34:35.978002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos -21269> 2013-04-03 10:34:35.978019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos version 393983 summary v 393983 -21268> 2013-04-03 10:34:35.978028 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21267> 2013-04-03 10:34:35.978039 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 preprocess_log log(1 entries) v1 from mon.0 -21266> 2013-04-03 10:34:35.978053 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21265> 2013-04-03 10:34:35.978065 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 prepare_log log(1 entries) v1 from mon.0 -21264> 2013-04-03 10:34:35.978074 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 logging 2013-04-03 10:34:35.977881 mon.0 10.252.24.25:6789/0 36 : [INF] pgmap v28978: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 7732B/s wr, 2op/s -21263> 2013-04-03 10:34:35.990609 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112996 ==== paxos(lease_ack lc 427236 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (2225950713 0 0) 0x2715780 con 0x208a840 -21262> 2013-04-03 10:34:35.990683 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos -21261> 2013-04-03 10:34:35.990702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393983 update_from_paxos version 393983 summary v 393983 -21260> 2013-04-03 10:34:35.990718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21259> 2013-04-03 10:34:36.922564 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112997 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 17 v187) v2) to leader v1 ==== 560+0+0 (1855502852 0 0) 0x2146080 con 0x208a840 -21258> 2013-04-03 10:34:36.922585 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -21257> 2013-04-03 10:34:36.922595 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae600 from 10.252.24.88:6789/0 -21256> 2013-04-03 10:34:36.922645 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 17 v187) v2 from mds.0 10.252.24.25:6802/3144 -21255> 2013-04-03 10:34:36.922656 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 17 v187) v2 tid 17414) v2 -- ?+0 0x20ad900 con 0x208a840 -21254> 2013-04-03 10:34:36.977541 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393983 encode_pending v393984 -21253> 2013-04-03 10:34:36.977573 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427223..427236) queue_proposal bl 443 bytes; ctx = 0x1b91a00 -21252> 2013-04-03 10:34:36.977581 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427223..427236) propose_queued 427237 443 bytes -21251> 2013-04-03 10:34:36.978498 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427236 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2715280 -21250> 2013-04-03 10:34:37.139708 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112998 ==== paxos(accept lc 427236 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (110679150 0 0) 0x2715280 con 0x208a840 -21249> 2013-04-03 10:34:37.140491 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427237 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2715780 -21248> 2013-04-03 10:34:37.140523 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427237 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2715500 -21247> 2013-04-03 10:34:37.140559 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 update_from_paxos -21246> 2013-04-03 10:34:37.140584 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 update_from_paxos version 393984 summary v 393983 -21245> 2013-04-03 10:34:37.140598 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 update_from_paxos latest full 393798 -21244> 2013-04-03 10:34:37.140616 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393984 update_from_paxos applying incremental log 393984 2013-04-03 10:34:35.977881 mon.0 10.252.24.25:6789/0 36 : [INF] pgmap v28978: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 7732B/s wr, 2op/s -21243> 2013-04-03 10:34:37.140660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 check_subs -21242> 2013-04-03 10:34:37.140689 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 create_pending v 393985 -21241> 2013-04-03 10:34:37.140710 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393984 _updated_log for mon.0 10.252.24.25:6789/0 -21240> 2013-04-03 10:34:37.140722 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 36) v1 -- ?+0 0x2224e00 con 0x1bac160 -21239> 2013-04-03 10:34:37.140769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 update_from_paxos -21238> 2013-04-03 10:34:37.140784 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 update_from_paxos version 393984 summary v 393984 -21237> 2013-04-03 10:34:37.140804 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21236> 2013-04-03 10:34:37.140824 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 36) v1 ==== 0+0+0 (0 0 0) 0x2224e00 con 0x1bac160 -21235> 2013-04-03 10:34:37.140832 7fcdffcdb700 10 handle_log_ack log(last 36) v1 -21234> 2013-04-03 10:34:37.140833 7fcdffcdb700 10 logged 2013-04-03 10:34:35.977881 mon.0 10.252.24.25:6789/0 36 : [INF] pgmap v28978: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 7732B/s wr, 2op/s -21233> 2013-04-03 10:34:37.164899 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698112999 ==== paxos(lease_ack lc 427237 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (3428673522 0 0) 0x2715500 con 0x208a840 -21232> 2013-04-03 10:34:37.164940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 update_from_paxos -21231> 2013-04-03 10:34:37.164956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393984 update_from_paxos version 393984 summary v 393984 -21230> 2013-04-03 10:34:37.164972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -21229> 2013-04-03 10:34:37.376772 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 14 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21673 v187) v2 ==== 289+0+0 (3517306455 0 0) 0x273ab00 con 0x200d420 -21228> 2013-04-03 10:34:37.376825 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21673 v187) v2 from mds.? 10.252.24.88:6800/17213 -21227> 2013-04-03 10:34:37.376836 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21673 v187) v2 -- ?+0 0x2146080 con 0x200d420 -21226> 2013-04-03 10:34:37.809417 7fcdfe8d5700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x26dc500 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept sd=24 10.252.24.30:45088/0 -21225> 2013-04-03 10:34:37.810087 7fcdfe8d5700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.30:6789/0 mon protocol 2 -21224> 2013-04-03 10:34:37.810202 7fcdfe8d5700 0 -- 10.252.24.25:6789/0 >> 10.252.24.30:6789/0 pipe(0x26dc500 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state connecting -21223> 2013-04-03 10:34:38.157254 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113000 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (424405211 0 0) 0x20e3dc0 con 0x208a840 -21222> 2013-04-03 10:34:38.157276 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -21221> 2013-04-03 10:34:38.157288 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -21220> 2013-04-03 10:34:38.157299 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 -21219> 2013-04-03 10:34:38.157304 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( start_reply flags( reply-to(mon.2 10.252.24.88:6789/0) ) ) v1 -- ?+0 0x273ab00 -21218> 2013-04-03 10:34:38.323598 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 check_down_pgs -21217> 2013-04-03 10:34:38.323627 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 v28978: 240 pgs: 240 active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 7732B/s wr, 2op/s -21216> 2013-04-03 10:34:38.323658 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -21215> 2013-04-03 10:34:38.323682 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 e599: 5 osds: 5 up, 5 in -21214> 2013-04-03 10:34:38.323719 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 min_last_epoch_clean 593 -21213> 2013-04-03 10:34:38.323749 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos -21212> 2013-04-03 10:34:38.323767 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos version 393984 summary v 393984 -21211> 2013-04-03 10:34:38.323776 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 log -21210> 2013-04-03 10:34:38.323793 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -21209> 2013-04-03 10:34:38.323811 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -21208> 2013-04-03 10:34:39.576743 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 15 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140303 v187) v2 ==== 289+0+0 (2718818576 0 0) 0x20e48c0 con 0x2118420 -21207> 2013-04-03 10:34:39.576812 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140303 v187) v2 from mds.? 10.252.58.31:6800/10908 -21206> 2013-04-03 10:34:39.576827 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140303 v187) v2 -- ?+0 0x20e3dc0 con 0x2118420 -21205> 2013-04-03 10:34:40.121251 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x26dc280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:32948/0 -21204> 2013-04-03 10:34:40.121289 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x26dc280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:32948/0) -21203> 2013-04-03 10:34:40.121306 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x26dc280 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -21202> 2013-04-03 10:34:40.121509 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x26dc280 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -21201> 2013-04-03 10:34:40.140591 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427237 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2715000 -21200> 2013-04-03 10:34:40.141399 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113001 ==== paxos(lease_ack lc 427237 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (2266830462 0 0) 0x2715000 con 0x208a840 -21199> 2013-04-03 10:34:40.141464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos -21198> 2013-04-03 10:34:40.141483 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos version 393984 summary v 393984 -21197> 2013-04-03 10:34:40.141502 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -21196> 2013-04-03 10:34:40.922601 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113002 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 18 v187) v2) to leader v1 ==== 560+0+0 (3983166053 0 0) 0x1bae340 con 0x208a840 -21195> 2013-04-03 10:34:40.922616 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -21194> 2013-04-03 10:34:40.922625 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273bb80 from 10.252.24.88:6789/0 -21193> 2013-04-03 10:34:40.922661 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 18 v187) v2 from mds.0 10.252.24.25:6802/3144 -21192> 2013-04-03 10:34:40.922673 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 18 v187) v2 tid 17415) v2 -- ?+0 0x20adb40 con 0x208a840 -21191> 2013-04-03 10:34:41.376897 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 15 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21674 v187) v2 ==== 289+0+0 (3021346768 0 0) 0x273a840 con 0x200d420 -21190> 2013-04-03 10:34:41.376967 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21674 v187) v2 from mds.? 10.252.24.88:6800/17213 -21189> 2013-04-03 10:34:41.376982 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21674 v187) v2 -- ?+0 0x1bae340 con 0x200d420 -21188> 2013-04-03 10:34:42.481929 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113003 ==== forward(osd_boot(osd.2 booted 0 v599) v3) to leader v1 ==== 898+0+0 (3204478712 0 0) 0x20e48c0 con 0x208a840 -21187> 2013-04-03 10:34:42.481949 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 -21186> 2013-04-03 10:34:42.481960 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2071800 from 10.252.24.88:6789/0 -21185> 2013-04-03 10:34:42.482009 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 preprocess_query osd_boot(osd.2 booted 0 v599) v3 from osd.2 10.252.24.30:6800/1508 -21184> 2013-04-03 10:34:42.482018 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 preprocess_boot from osd.2 10.252.24.30:6800/1508 -21183> 2013-04-03 10:34:42.482023 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 prepare_update osd_boot(osd.2 booted 0 v599) v3 from osd.2 10.252.24.30:6800/1508 -21182> 2013-04-03 10:34:42.482027 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 prepare_boot from osd.2 10.252.24.30:6800/1508 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.2 eb9ff30e-cc75-49d3-8ef9-2f478e173371 e599 [98,599] lci=[593,599]) cluster_addr 10.252.24.30:6804/1508 hb_addr 10.252.24.30:6805/1508 -21181> 2013-04-03 10:34:42.482038 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 prepare_boot was up, first marking down osd.2 10.252.24.30:6801/10373 -21180> 2013-04-03 10:34:42.482042 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 should_propose -21179> 2013-04-03 10:34:42.532195 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 min_last_epoch_clean 593 -21178> 2013-04-03 10:34:42.532253 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 min_last_epoch_clean 593 -21177> 2013-04-03 10:34:42.532306 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 encode_pending e 600 -21176> 2013-04-03 10:34:42.532311 7fce004dc700 2 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 osd.2 DOWN -21175> 2013-04-03 10:34:42.532343 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427237) queue_proposal bl 289 bytes; ctx = 0x1b919f0 -21174> 2013-04-03 10:34:42.532352 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427237) propose_queued 427238 289 bytes -21173> 2013-04-03 10:34:42.533299 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427237 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dcc80 -21172> 2013-04-03 10:34:42.567475 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113004 ==== paxos(accept lc 427237 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3975180938 0 0) 0x26dcc80 con 0x208a840 -21171> 2013-04-03 10:34:42.567785 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427238 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2715000 -21170> 2013-04-03 10:34:42.567817 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427238 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2715500 -21169> 2013-04-03 10:34:42.567888 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e599 update_from_paxos applying incremental 600 -21168> 2013-04-03 10:34:42.567932 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 e600: 5 osds: 4 up, 5 in -21167> 2013-04-03 10:34:42.568912 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 adding osd.2 to down_pending_out map -21166> 2013-04-03 10:34:42.568965 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 check_osd_map applying osdmap e600 to pg_map -21165> 2013-04-03 10:34:42.568984 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 register_new_pgs checking pg pools for osdmap epoch 600, last_pg_scan 341 -21164> 2013-04-03 10:34:42.568989 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 -21163> 2013-04-03 10:34:42.568993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -21162> 2013-04-03 10:34:42.568996 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -21161> 2013-04-03 10:34:42.568999 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 -21160> 2013-04-03 10:34:42.569002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 -21159> 2013-04-03 10:34:42.569005 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 -21158> 2013-04-03 10:34:42.569007 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 -21157> 2013-04-03 10:34:42.569010 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 -21156> 2013-04-03 10:34:42.569013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 -21155> 2013-04-03 10:34:42.569023 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs -21154> 2013-04-03 10:34:42.569025 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 check_down_pgs -21153> 2013-04-03 10:34:42.569027 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.d stale with acting [2,0] -21152> 2013-04-03 10:34:42.569034 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 8.3 stale with acting [2,3] -21151> 2013-04-03 10:34:42.569037 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.9 stale with acting [2,3] -21150> 2013-04-03 10:34:42.569041 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.a stale with acting [2,3] -21149> 2013-04-03 10:34:42.569044 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.b stale with acting [2,3] -21148> 2013-04-03 10:34:42.569047 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 7.0 stale with acting [2,3] -21147> 2013-04-03 10:34:42.569050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 6.1 stale with acting [2,3] -21146> 2013-04-03 10:34:42.569053 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 5.2 stale with acting [2,3] -21145> 2013-04-03 10:34:42.569055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 4.3 stale with acting [2,3] -21144> 2013-04-03 10:34:42.569058 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 3.4 stale with acting [2,3] -21143> 2013-04-03 10:34:42.569061 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.5 stale with acting [2,3] -21142> 2013-04-03 10:34:42.569064 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.6 stale with acting [2,3] -21141> 2013-04-03 10:34:42.569067 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.7 stale with acting [2,3] -21140> 2013-04-03 10:34:42.569070 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 5.0 stale with acting [2,0] -21139> 2013-04-03 10:34:42.569072 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 4.1 stale with acting [2,0] -21138> 2013-04-03 10:34:42.569075 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.4 stale with acting [2,0] -21137> 2013-04-03 10:34:42.569078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.5 stale with acting [2,0] -21136> 2013-04-03 10:34:42.569081 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 7.4 stale with acting [2,3] -21135> 2013-04-03 10:34:42.569084 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 6.5 stale with acting [2,3] -21134> 2013-04-03 10:34:42.569087 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 5.6 stale with acting [2,3] -21133> 2013-04-03 10:34:42.569091 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 4.7 stale with acting [2,3] -21132> 2013-04-03 10:34:42.569093 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.1 stale with acting [2,4] -21131> 2013-04-03 10:34:42.569096 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.2 stale with acting [2,4] -21130> 2013-04-03 10:34:42.569099 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.3 stale with acting [2,4] -21129> 2013-04-03 10:34:42.569102 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 3.0 stale with acting [2,4] -21128> 2013-04-03 10:34:42.569105 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 3.2 stale with acting [2,0] -21127> 2013-04-03 10:34:42.569113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.3 stale with acting [2,0] -21126> 2013-04-03 10:34:42.569116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 7.7 stale with acting [2,0] -21125> 2013-04-03 10:34:42.569120 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.3c stale with acting [2,3] -21124> 2013-04-03 10:34:42.569123 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.3e stale with acting [2,3] -21123> 2013-04-03 10:34:42.569126 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.3d stale with acting [2,3] -21122> 2013-04-03 10:34:42.569129 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.39 stale with acting [2,0] -21121> 2013-04-03 10:34:42.569132 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.3a stale with acting [2,0] -21120> 2013-04-03 10:34:42.569135 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.3b stale with acting [2,0] -21119> 2013-04-03 10:34:42.569138 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.38 stale with acting [2,3] -21118> 2013-04-03 10:34:42.569141 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.3a stale with acting [2,3] -21117> 2013-04-03 10:34:42.569144 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.39 stale with acting [2,3] -21116> 2013-04-03 10:34:42.569147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.34 stale with acting [2,1] -21115> 2013-04-03 10:34:42.569150 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.36 stale with acting [2,1] -21114> 2013-04-03 10:34:42.569156 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.35 stale with acting [2,1] -21113> 2013-04-03 10:34:42.569159 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.29 stale with acting [2,0] -21112> 2013-04-03 10:34:42.569162 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.2a stale with acting [2,0] -21111> 2013-04-03 10:34:42.569165 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.2b stale with acting [2,0] -21110> 2013-04-03 10:34:42.569168 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.28 stale with acting [2,0] -21109> 2013-04-03 10:34:42.569171 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.2a stale with acting [2,0] -21108> 2013-04-03 10:34:42.569173 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.28 stale with acting [2,1] -21107> 2013-04-03 10:34:42.569176 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.29 stale with acting [2,1] -21106> 2013-04-03 10:34:42.569179 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.29 stale with acting [2,0] -21105> 2013-04-03 10:34:42.569182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.27 stale with acting [2,1] -21104> 2013-04-03 10:34:42.569185 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.21 stale with acting [2,3] -21103> 2013-04-03 10:34:42.569188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.22 stale with acting [2,3] -21102> 2013-04-03 10:34:42.569191 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.23 stale with acting [2,3] -21101> 2013-04-03 10:34:42.569194 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.20 stale with acting [2,1] -21100> 2013-04-03 10:34:42.569197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.22 stale with acting [2,1] -21099> 2013-04-03 10:34:42.569199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.21 stale with acting [2,1] -21098> 2013-04-03 10:34:42.569203 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.1c stale with acting [2,3] -21097> 2013-04-03 10:34:42.569205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.1b stale with acting [2,3] -21096> 2013-04-03 10:34:42.569208 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.19 stale with acting [2,3] -21095> 2013-04-03 10:34:42.569214 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.18 stale with acting [2,3] -21094> 2013-04-03 10:34:42.569217 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.18 stale with acting [2,3] -21093> 2013-04-03 10:34:42.569220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.1a stale with acting [2,3] -21092> 2013-04-03 10:34:42.569223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.17 stale with acting [2,3] -21091> 2013-04-03 10:34:42.569229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.17 stale with acting [2,3] -21090> 2013-04-03 10:34:42.569232 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.16 stale with acting [2,3] -21089> 2013-04-03 10:34:42.569235 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.f stale with acting [2,3] -21088> 2013-04-03 10:34:42.569238 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 1.e stale with acting [2,3] -21087> 2013-04-03 10:34:42.569241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.d stale with acting [2,3] -21086> 2013-04-03 10:34:42.569244 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 8.7 stale with acting [2,3] -21085> 2013-04-03 10:34:42.569247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 0.e stale with acting [2,0] -21084> 2013-04-03 10:34:42.569249 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 2.c stale with acting [2,0] -21083> 2013-04-03 10:34:42.569252 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 marking pg 8.6 stale with acting [2,0] -21082> 2013-04-03 10:34:42.569305 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 encode_pending v 28979 -21081> 2013-04-03 10:34:42.569514 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427238) queue_proposal bl 26698 bytes; ctx = 0x1b916d0 -21080> 2013-04-03 10:34:42.569523 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427238) propose_queued 427239 26698 bytes -21079> 2013-04-03 10:34:42.571055 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427238 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2715280 -21078> 2013-04-03 10:34:42.571087 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 send_pg_creates to 0 pgs -21077> 2013-04-03 10:34:42.571093 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 send_to_waiting 600 -21076> 2013-04-03 10:34:42.571100 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 share_map_with_random_osd no up osd on our session map -21075> 2013-04-03 10:34:42.571102 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 update_logger -21074> 2013-04-03 10:34:42.571122 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 create_pending e 601 -21073> 2013-04-03 10:34:42.571125 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 remove_redundant_pg_temp -21072> 2013-04-03 10:34:42.571127 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 remove_down_pg_temp -21071> 2013-04-03 10:34:42.571160 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_query osd_boot(osd.2 booted 0 v599) v3 from osd.2 10.252.24.30:6800/1508 -21070> 2013-04-03 10:34:42.571169 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_boot from osd.2 10.252.24.30:6800/1508 -21069> 2013-04-03 10:34:42.571173 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_update osd_boot(osd.2 booted 0 v599) v3 from osd.2 10.252.24.30:6800/1508 -21068> 2013-04-03 10:34:42.571176 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_boot from osd.2 10.252.24.30:6800/1508 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.2 eb9ff30e-cc75-49d3-8ef9-2f478e173371 e599 [98,599] lci=[593,599]) cluster_addr 10.252.24.30:6804/1508 hb_addr 10.252.24.30:6805/1508 -21067> 2013-04-03 10:34:42.571188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 setting osd.2 uuid to eb9ff30e-cc75-49d3-8ef9-2f478e173371 -21066> 2013-04-03 10:34:42.571192 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 old osd_info: up_from 593 up_thru 598 down_at 600 last_clean_interval [586,590) lost_at 95 -21065> 2013-04-03 10:34:42.571195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_boot osd.2 last_clean_interval [586,590) -> [593-599) -21064> 2013-04-03 10:34:42.571198 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 not laggy, new xi down_stamp 2013-04-03 10:34:42.532310 laggy_probability 0 laggy_interval 0 -21063> 2013-04-03 10:34:42.571208 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 should_propose -21062> 2013-04-03 10:34:42.571220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 update_logger -21061> 2013-04-03 10:34:42.571227 7fcdffcdb700 0 log [INF] : osdmap e600: 5 osds: 4 up, 5 in -21060> 2013-04-03 10:34:42.571228 7fcdffcdb700 10 send_log to self -21059> 2013-04-03 10:34:42.571230 7fcdffcdb700 10 log_queue is 1 last_log 37 sent 36 num 1 unsent 1 sending 1 -21058> 2013-04-03 10:34:42.571232 7fcdffcdb700 10 will send 2013-04-03 10:34:42.571227 mon.0 10.252.24.25:6789/0 37 : [INF] osdmap e600: 5 osds: 4 up, 5 in -21057> 2013-04-03 10:34:42.571244 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2227400 -21056> 2013-04-03 10:34:42.571271 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2227400 con 0x1bac160 -21055> 2013-04-03 10:34:42.571295 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos -21054> 2013-04-03 10:34:42.571311 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos version 393984 summary v 393984 -21053> 2013-04-03 10:34:42.571321 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21052> 2013-04-03 10:34:42.571333 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 preprocess_log log(1 entries) v1 from mon.0 -21051> 2013-04-03 10:34:42.571351 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21050> 2013-04-03 10:34:42.571364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 prepare_log log(1 entries) v1 from mon.0 -21049> 2013-04-03 10:34:42.571376 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 logging 2013-04-03 10:34:42.571227 mon.0 10.252.24.25:6789/0 37 : [INF] osdmap e600: 5 osds: 4 up, 5 in -21048> 2013-04-03 10:34:42.592666 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113005 ==== paxos(lease_ack lc 427238 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (2448883084 0 0) 0x2715280 con 0x208a840 -21047> 2013-04-03 10:34:42.642950 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113006 ==== paxos(accept lc 427238 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4071412633 0 0) 0x2715500 con 0x208a840 -21046> 2013-04-03 10:34:42.689980 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427239 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2715280 -21045> 2013-04-03 10:34:42.689998 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427239 fc 427223 pn 0 opn 0) v3 -- ?+0 0x26dcc80 -21044> 2013-04-03 10:34:42.690096 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28978 update_from_paxos applying incremental 28979 -21043> 2013-04-03 10:34:42.690270 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 v28979: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 4526B/s wr, 1op/s -21042> 2013-04-03 10:34:42.690300 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 send_pg_creates to 0 pgs -21041> 2013-04-03 10:34:42.690304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 update_logger -21040> 2013-04-03 10:34:42.690323 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 create_pending v 28980 -21039> 2013-04-03 10:34:42.690334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 check_osd_map already seen 600 >= 600 -21038> 2013-04-03 10:34:42.690339 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 update_logger -21037> 2013-04-03 10:34:42.690350 7fcdffcdb700 0 log [INF] : pgmap v28979: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 4526B/s wr, 1op/s -21036> 2013-04-03 10:34:42.690351 7fcdffcdb700 10 send_log to self -21035> 2013-04-03 10:34:42.690352 7fcdffcdb700 10 log_queue is 2 last_log 38 sent 37 num 2 unsent 1 sending 1 -21034> 2013-04-03 10:34:42.690353 7fcdffcdb700 10 will send 2013-04-03 10:34:42.571227 mon.0 10.252.24.25:6789/0 37 : [INF] osdmap e600: 5 osds: 4 up, 5 in -21033> 2013-04-03 10:34:42.690360 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2544e00 -21032> 2013-04-03 10:34:42.690404 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos -21031> 2013-04-03 10:34:42.690421 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos version 393984 summary v 393984 -21030> 2013-04-03 10:34:42.690439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -21029> 2013-04-03 10:34:42.690461 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2544e00 con 0x1bac160 -21028> 2013-04-03 10:34:42.690482 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos -21027> 2013-04-03 10:34:42.690497 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos version 393984 summary v 393984 -21026> 2013-04-03 10:34:42.690507 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -21025> 2013-04-03 10:34:42.690518 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 preprocess_log log(1 entries) v1 from mon.0 -21024> 2013-04-03 10:34:42.690533 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 nothing new -21023> 2013-04-03 10:34:42.766702 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113007 ==== paxos(lease_ack lc 427239 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (2708212771 0 0) 0x26dcc80 con 0x208a840 -21022> 2013-04-03 10:34:42.766751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos -21021> 2013-04-03 10:34:42.766768 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos version 393984 summary v 393984 -21020> 2013-04-03 10:34:42.766784 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -21019> 2013-04-03 10:34:42.766804 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113008 ==== forward(osd_alive(want up_thru 600 have 600) v1) to leader v1 ==== 294+0+0 (1004601684 0 0) 0x273b8c0 con 0x208a840 -21018> 2013-04-03 10:34:42.766814 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -21017> 2013-04-03 10:34:42.766822 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x22248c0 from 10.252.24.88:6789/0 -21016> 2013-04-03 10:34:42.766845 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_query osd_alive(want up_thru 600 have 600) v1 from osd.1 10.252.24.25:6805/3572 -21015> 2013-04-03 10:34:42.766852 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_alive want up_thru 600 from osd.1 10.252.24.25:6805/3572 -21014> 2013-04-03 10:34:42.766857 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_update osd_alive(want up_thru 600 have 600) v1 from osd.1 10.252.24.25:6805/3572 -21013> 2013-04-03 10:34:42.766860 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_alive want up_thru 600 have 600 from osd.1 10.252.24.25:6805/3572 -21012> 2013-04-03 10:34:42.766864 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 should_propose -21011> 2013-04-03 10:34:42.766877 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113009 ==== forward(osd_alive(want up_thru 600 have 600) v1) to leader v1 ==== 294+0+0 (637724634 0 0) 0x273b600 con 0x208a840 -21010> 2013-04-03 10:34:42.766886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 -21009> 2013-04-03 10:34:42.766893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x22241c0 from 10.252.24.88:6789/0 -21008> 2013-04-03 10:34:42.766915 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_query osd_alive(want up_thru 600 have 600) v1 from osd.4 10.252.24.32:6800/13473 -21007> 2013-04-03 10:34:42.766921 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_alive want up_thru 600 from osd.4 10.252.24.32:6800/13473 -21006> 2013-04-03 10:34:42.766925 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_update osd_alive(want up_thru 600 have 600) v1 from osd.4 10.252.24.32:6800/13473 -21005> 2013-04-03 10:34:42.766928 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_alive want up_thru 600 have 600 from osd.4 10.252.24.32:6800/13473 -21004> 2013-04-03 10:34:42.766931 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 should_propose -21003> 2013-04-03 10:34:42.766941 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113010 ==== forward(osd_alive(want up_thru 600 have 600) v1) to leader v1 ==== 294+0+0 (3723063016 0 0) 0x273b340 con 0x208a840 -21002> 2013-04-03 10:34:42.766947 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -21001> 2013-04-03 10:34:42.766952 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2224c40 from 10.252.24.88:6789/0 -21000> 2013-04-03 10:34:42.766974 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_query osd_alive(want up_thru 600 have 600) v1 from osd.3 10.252.24.88:6801/17538 -20999> 2013-04-03 10:34:42.766981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_alive want up_thru 600 from osd.3 10.252.24.88:6801/17538 -20998> 2013-04-03 10:34:42.766985 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_update osd_alive(want up_thru 600 have 600) v1 from osd.3 10.252.24.88:6801/17538 -20997> 2013-04-03 10:34:42.766988 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_alive want up_thru 600 have 600 from osd.3 10.252.24.88:6801/17538 -20996> 2013-04-03 10:34:42.766991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 should_propose -20995> 2013-04-03 10:34:42.767001 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113011 ==== forward(osd_alive(want up_thru 600 have 600) v1) to leader v1 ==== 294+0+0 (1276377059 0 0) 0x273b080 con 0x208a840 -20994> 2013-04-03 10:34:42.767007 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -20993> 2013-04-03 10:34:42.767013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2224540 from 10.252.24.88:6789/0 -20992> 2013-04-03 10:34:42.767033 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_query osd_alive(want up_thru 600 have 600) v1 from osd.0 10.252.58.31:6802/2682 -20991> 2013-04-03 10:34:42.767040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 preprocess_alive want up_thru 600 from osd.0 10.252.58.31:6802/2682 -20990> 2013-04-03 10:34:42.767044 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_update osd_alive(want up_thru 600 have 600) v1 from osd.0 10.252.58.31:6802/2682 -20989> 2013-04-03 10:34:42.767047 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 prepare_alive want up_thru 600 have 600 from osd.0 10.252.58.31:6802/2682 -20988> 2013-04-03 10:34:42.767050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 should_propose -20987> 2013-04-03 10:34:43.323962 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 check_down_pgs -20986> 2013-04-03 10:34:43.323988 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 v28979: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 4526B/s wr, 1op/s -20985> 2013-04-03 10:34:43.324016 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -20984> 2013-04-03 10:34:43.324039 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 e600: 5 osds: 4 up, 5 in -20983> 2013-04-03 10:34:43.324076 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 min_last_epoch_clean 593 -20982> 2013-04-03 10:34:43.324106 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos -20981> 2013-04-03 10:34:43.324123 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 update_from_paxos version 393984 summary v 393984 -20980> 2013-04-03 10:34:43.324131 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 log -20979> 2013-04-03 10:34:43.324147 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20978> 2013-04-03 10:34:43.324166 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -20977> 2013-04-03 10:34:43.567657 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393984 encode_pending v393985 -20976> 2013-04-03 10:34:43.567688 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427239) queue_proposal bl 364 bytes; ctx = 0x1b918c0 -20975> 2013-04-03 10:34:43.567697 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427239) propose_queued 427240 364 bytes -20974> 2013-04-03 10:34:43.569228 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427239 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dca00 -20973> 2013-04-03 10:34:43.569299 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 min_last_epoch_clean 593 -20972> 2013-04-03 10:34:43.569358 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 min_last_epoch_clean 593 -20971> 2013-04-03 10:34:43.569407 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 encode_pending e 601 -20970> 2013-04-03 10:34:43.569411 7fce004dc700 2 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 osd.2 UP 10.252.24.30:6800/1508 -20969> 2013-04-03 10:34:43.569443 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427223..427239) queue_proposal bl 774 bytes; ctx = 0x1b919e0 -20968> 2013-04-03 10:34:43.569450 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427223..427239) propose_new_value not active; proposal queued -20967> 2013-04-03 10:34:43.576856 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 16 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140304 v187) v2 ==== 289+0+0 (1773029960 0 0) 0x1baeb80 con 0x2118420 -20966> 2013-04-03 10:34:43.576902 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140304 v187) v2 from mds.? 10.252.58.31:6800/10908 -20965> 2013-04-03 10:34:43.576913 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140304 v187) v2 -- ?+0 0x273b080 con 0x2118420 -20964> 2013-04-03 10:34:43.832843 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113012 ==== paxos(accept lc 427239 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (273785865 0 0) 0x26dca00 con 0x208a840 -20963> 2013-04-03 10:34:43.833922 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427240 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dcc80 -20962> 2013-04-03 10:34:43.833954 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427240 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2715500 -20961> 2013-04-03 10:34:43.833997 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos -20960> 2013-04-03 10:34:43.834023 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos version 393985 summary v 393984 -20959> 2013-04-03 10:34:43.834037 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos latest full 393798 -20958> 2013-04-03 10:34:43.834056 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos applying incremental log 393985 2013-04-03 10:34:42.571227 mon.0 10.252.24.25:6789/0 37 : [INF] osdmap e600: 5 osds: 4 up, 5 in -20957> 2013-04-03 10:34:43.834097 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 check_subs -20956> 2013-04-03 10:34:43.834127 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 create_pending v 393986 -20955> 2013-04-03 10:34:43.834149 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 _updated_log for mon.0 10.252.24.25:6789/0 -20954> 2013-04-03 10:34:43.834162 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 37) v1 -- ?+0 0x2224e00 con 0x1bac160 -20953> 2013-04-03 10:34:43.834186 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427240) propose_queued 427241 774 bytes -20952> 2013-04-03 10:34:43.834639 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427240 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787c80 -20951> 2013-04-03 10:34:43.834681 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 37) v1 ==== 0+0+0 (0 0 0) 0x2224e00 con 0x1bac160 -20950> 2013-04-03 10:34:43.834690 7fcdffcdb700 10 handle_log_ack log(last 37) v1 -20949> 2013-04-03 10:34:43.834693 7fcdffcdb700 10 logged 2013-04-03 10:34:42.571227 mon.0 10.252.24.25:6789/0 37 : [INF] osdmap e600: 5 osds: 4 up, 5 in -20948> 2013-04-03 10:34:43.858024 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113013 ==== paxos(lease_ack lc 427240 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (665659021 0 0) 0x2787c80 con 0x208a840 -20947> 2013-04-03 10:34:43.871185 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113014 ==== paxos(accept lc 427240 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2072372470 0 0) 0x2715500 con 0x208a840 -20946> 2013-04-03 10:34:43.871467 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427241 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787c80 -20945> 2013-04-03 10:34:43.871497 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427241 fc 427223 pn 0 opn 0) v3 -- ?+0 0x26dca00 -20944> 2013-04-03 10:34:43.871564 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e600 update_from_paxos applying incremental 601 -20943> 2013-04-03 10:34:43.871612 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 e601: 5 osds: 5 up, 5 in -20942> 2013-04-03 10:34:43.872723 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 check_osd_map applying osdmap e601 to pg_map -20941> 2013-04-03 10:34:43.872751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 register_new_pgs checking pg pools for osdmap epoch 601, last_pg_scan 341 -20940> 2013-04-03 10:34:43.872754 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 -20939> 2013-04-03 10:34:43.872758 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -20938> 2013-04-03 10:34:43.872762 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -20937> 2013-04-03 10:34:43.872765 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 -20936> 2013-04-03 10:34:43.872768 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 -20935> 2013-04-03 10:34:43.872771 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 -20934> 2013-04-03 10:34:43.872774 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 -20933> 2013-04-03 10:34:43.872783 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 -20932> 2013-04-03 10:34:43.872786 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 -20931> 2013-04-03 10:34:43.872795 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs -20930> 2013-04-03 10:34:43.872844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 encode_pending v 28980 -20929> 2013-04-03 10:34:43.872875 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427241) queue_proposal bl 215 bytes; ctx = 0x1b919b0 -20928> 2013-04-03 10:34:43.872883 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427241) propose_queued 427242 215 bytes -20927> 2013-04-03 10:34:43.873081 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427241 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787a00 -20926> 2013-04-03 10:34:43.873114 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 send_pg_creates to 0 pgs -20925> 2013-04-03 10:34:43.873128 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_to_waiting 601 -20924> 2013-04-03 10:34:43.873133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 share_map_with_random_osd no up osd on our session map -20923> 2013-04-03 10:34:43.873140 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 update_logger -20922> 2013-04-03 10:34:43.873169 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 create_pending e 602 -20921> 2013-04-03 10:34:43.873174 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 remove_redundant_pg_temp -20920> 2013-04-03 10:34:43.873176 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 remove_down_pg_temp -20919> 2013-04-03 10:34:43.873195 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 _booted osd.2 10.252.24.30:6800/1508 w 0 from 599 -20918> 2013-04-03 10:34:43.873212 7fcdffcdb700 0 log [INF] : osd.2 10.252.24.30:6800/1508 boot -20917> 2013-04-03 10:34:43.873214 7fcdffcdb700 10 send_log to self -20916> 2013-04-03 10:34:43.873214 7fcdffcdb700 10 log_queue is 2 last_log 39 sent 37 num 2 unsent 2 sending 2 -20915> 2013-04-03 10:34:43.873216 7fcdffcdb700 10 will send 2013-04-03 10:34:42.690351 mon.0 10.252.24.25:6789/0 38 : [INF] pgmap v28979: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 4526B/s wr, 1op/s -20914> 2013-04-03 10:34:43.873221 7fcdffcdb700 10 will send 2013-04-03 10:34:43.873213 mon.0 10.252.24.25:6789/0 39 : [INF] osd.2 10.252.24.30:6800/1508 boot -20913> 2013-04-03 10:34:43.873226 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2227400 -20912> 2013-04-03 10:34:43.873247 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_latest to osd.2 10.252.24.30:6800/1508 start 600 -20911> 2013-04-03 10:34:43.873254 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_incremental [600..601] to osd.2 10.252.24.30:6800/1508 -20910> 2013-04-03 10:34:43.873266 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 build_incremental [600..601] -20909> 2013-04-03 10:34:43.873300 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(600..601 src has 100..601) v3 tid 17416) v2 -- ?+0 0x20ad900 con 0x208a840 -20908> 2013-04-03 10:34:43.873313 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 _reply_map 600 from osd.1 10.252.24.25:6805/3572 -20907> 2013-04-03 10:34:43.873331 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_latest to osd.1 10.252.24.25:6805/3572 start 600 -20906> 2013-04-03 10:34:43.873337 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_incremental [600..601] to osd.1 10.252.24.25:6805/3572 -20905> 2013-04-03 10:34:43.873348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 build_incremental [600..601] -20904> 2013-04-03 10:34:43.873380 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(600..601 src has 100..601) v3 tid 17417) v2 -- ?+0 0x20fcd80 con 0x208a840 -20903> 2013-04-03 10:34:43.873390 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 _reply_map 600 from osd.4 10.252.24.32:6800/13473 -20902> 2013-04-03 10:34:43.873409 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_latest to osd.4 10.252.24.32:6800/13473 start 600 -20901> 2013-04-03 10:34:43.873414 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_incremental [600..601] to osd.4 10.252.24.32:6800/13473 -20900> 2013-04-03 10:34:43.873425 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 build_incremental [600..601] -20899> 2013-04-03 10:34:43.873456 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(600..601 src has 100..601) v3 tid 17418) v2 -- ?+0 0x20ad240 con 0x208a840 -20898> 2013-04-03 10:34:43.873465 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 _reply_map 600 from osd.3 10.252.24.88:6801/17538 -20897> 2013-04-03 10:34:43.873489 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_latest to osd.3 10.252.24.88:6801/17538 start 600 -20896> 2013-04-03 10:34:43.873495 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_incremental [600..601] to osd.3 10.252.24.88:6801/17538 -20895> 2013-04-03 10:34:43.873506 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 build_incremental [600..601] -20894> 2013-04-03 10:34:43.873538 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(600..601 src has 100..601) v3 tid 17419) v2 -- ?+0 0x20ad6c0 con 0x208a840 -20893> 2013-04-03 10:34:43.873548 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 _reply_map 600 from osd.0 10.252.58.31:6802/2682 -20892> 2013-04-03 10:34:43.873568 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_latest to osd.0 10.252.58.31:6802/2682 start 600 -20891> 2013-04-03 10:34:43.873573 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 send_incremental [600..601] to osd.0 10.252.58.31:6802/2682 -20890> 2013-04-03 10:34:43.873584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 build_incremental [600..601] -20889> 2013-04-03 10:34:43.873617 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(600..601 src has 100..601) v3 tid 17420) v2 -- ?+0 0x24f8d80 con 0x208a840 -20888> 2013-04-03 10:34:43.873626 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 update_logger -20887> 2013-04-03 10:34:43.873633 7fcdffcdb700 0 log [INF] : osdmap e601: 5 osds: 5 up, 5 in -20886> 2013-04-03 10:34:43.873634 7fcdffcdb700 10 send_log to self -20885> 2013-04-03 10:34:43.873639 7fcdffcdb700 10 log_queue is 3 last_log 40 sent 39 num 3 unsent 1 sending 1 -20884> 2013-04-03 10:34:43.873641 7fcdffcdb700 10 will send 2013-04-03 10:34:43.873213 mon.0 10.252.24.25:6789/0 39 : [INF] osd.2 10.252.24.30:6800/1508 boot -20883> 2013-04-03 10:34:43.873646 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2055000 -20882> 2013-04-03 10:34:43.873673 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2227400 con 0x1bac160 -20881> 2013-04-03 10:34:43.873694 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos -20880> 2013-04-03 10:34:43.873710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos version 393985 summary v 393985 -20879> 2013-04-03 10:34:43.873719 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -20878> 2013-04-03 10:34:43.873731 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 preprocess_log log(2 entries) v1 from mon.0 -20877> 2013-04-03 10:34:43.873749 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -20876> 2013-04-03 10:34:43.873762 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 prepare_log log(2 entries) v1 from mon.0 -20875> 2013-04-03 10:34:43.873773 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 logging 2013-04-03 10:34:42.690351 mon.0 10.252.24.25:6789/0 38 : [INF] pgmap v28979: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 4526B/s wr, 1op/s -20874> 2013-04-03 10:34:43.873791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 logging 2013-04-03 10:34:43.873213 mon.0 10.252.24.25:6789/0 39 : [INF] osd.2 10.252.24.30:6800/1508 boot -20873> 2013-04-03 10:34:43.873827 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2055000 con 0x1bac160 -20872> 2013-04-03 10:34:43.873848 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos -20871> 2013-04-03 10:34:43.873863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos version 393985 summary v 393985 -20870> 2013-04-03 10:34:43.873873 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -20869> 2013-04-03 10:34:43.873884 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 preprocess_log log(1 entries) v1 from mon.0 -20868> 2013-04-03 10:34:43.873899 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 nothing new -20867> 2013-04-03 10:34:43.883167 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113015 ==== paxos(lease_ack lc 427241 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (183694454 0 0) 0x2799c80 con 0x208a840 -20866> 2013-04-03 10:34:43.920447 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113016 ==== paxos(accept lc 427241 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1165171711 0 0) 0x2787000 con 0x208a840 -20865> 2013-04-03 10:34:43.923098 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427242 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2799c80 -20864> 2013-04-03 10:34:43.923131 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427242 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2715500 -20863> 2013-04-03 10:34:43.923198 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28979 update_from_paxos applying incremental 28980 -20862> 2013-04-03 10:34:43.923207 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 v28980: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail -20861> 2013-04-03 10:34:43.923224 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 send_pg_creates to 0 pgs -20860> 2013-04-03 10:34:43.923226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 update_logger -20859> 2013-04-03 10:34:43.923235 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 create_pending v 28981 -20858> 2013-04-03 10:34:43.923244 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 check_osd_map already seen 601 >= 601 -20857> 2013-04-03 10:34:43.923247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 update_logger -20856> 2013-04-03 10:34:43.923256 7fcdffcdb700 0 log [INF] : pgmap v28980: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail -20855> 2013-04-03 10:34:43.923258 7fcdffcdb700 10 send_log to self -20854> 2013-04-03 10:34:43.923258 7fcdffcdb700 10 log_queue is 4 last_log 41 sent 39 num 4 unsent 2 sending 2 -20853> 2013-04-03 10:34:43.923260 7fcdffcdb700 10 will send 2013-04-03 10:34:43.873213 mon.0 10.252.24.25:6789/0 39 : [INF] osd.2 10.252.24.30:6800/1508 boot -20852> 2013-04-03 10:34:43.923267 7fcdffcdb700 10 will send 2013-04-03 10:34:43.873634 mon.0 10.252.24.25:6789/0 40 : [INF] osdmap e601: 5 osds: 5 up, 5 in -20851> 2013-04-03 10:34:43.923284 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2055000 -20850> 2013-04-03 10:34:43.923330 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos -20849> 2013-04-03 10:34:43.923346 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos version 393985 summary v 393985 -20848> 2013-04-03 10:34:43.923364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20847> 2013-04-03 10:34:43.923385 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2055000 con 0x1bac160 -20846> 2013-04-03 10:34:43.923406 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos -20845> 2013-04-03 10:34:43.923421 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos version 393985 summary v 393985 -20844> 2013-04-03 10:34:43.923430 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -20843> 2013-04-03 10:34:43.923442 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 preprocess_log log(2 entries) v1 from mon.0 -20842> 2013-04-03 10:34:43.923461 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -20841> 2013-04-03 10:34:43.923473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 prepare_log log(2 entries) v1 from mon.0 -20840> 2013-04-03 10:34:43.923484 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 logging 2013-04-03 10:34:43.873213 mon.0 10.252.24.25:6789/0 39 : [INF] osd.2 10.252.24.30:6800/1508 boot -20839> 2013-04-03 10:34:43.923501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 logging 2013-04-03 10:34:43.873634 mon.0 10.252.24.25:6789/0 40 : [INF] osdmap e601: 5 osds: 5 up, 5 in -20838> 2013-04-03 10:34:43.958772 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113017 ==== paxos(lease_ack lc 427242 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (4057033603 0 0) 0x2715500 con 0x208a840 -20837> 2013-04-03 10:34:43.958817 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos -20836> 2013-04-03 10:34:43.958834 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 update_from_paxos version 393985 summary v 393985 -20835> 2013-04-03 10:34:43.958850 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20834> 2013-04-03 10:34:43.958868 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113018 ==== forward(osd_alive(want up_thru 601 have 601) v1) to leader v1 ==== 294+0+0 (1410140798 0 0) 0x273adc0 con 0x208a840 -20833> 2013-04-03 10:34:43.958877 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -20832> 2013-04-03 10:34:43.958885 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2224380 from 10.252.24.88:6789/0 -20831> 2013-04-03 10:34:43.958909 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_query osd_alive(want up_thru 601 have 601) v1 from osd.1 10.252.24.25:6805/3572 -20830> 2013-04-03 10:34:43.958917 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_alive want up_thru 601 from osd.1 10.252.24.25:6805/3572 -20829> 2013-04-03 10:34:43.958921 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_update osd_alive(want up_thru 601 have 601) v1 from osd.1 10.252.24.25:6805/3572 -20828> 2013-04-03 10:34:43.958926 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_alive want up_thru 601 have 601 from osd.1 10.252.24.25:6805/3572 -20827> 2013-04-03 10:34:43.958930 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 should_propose -20826> 2013-04-03 10:34:43.958941 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113019 ==== forward(osd_alive(want up_thru 601 have 601) v1) to leader v1 ==== 294+0+0 (2518116518 0 0) 0x273b8c0 con 0x208a840 -20825> 2013-04-03 10:34:43.958948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 -20824> 2013-04-03 10:34:43.958953 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2294380 from 10.252.24.88:6789/0 -20823> 2013-04-03 10:34:43.958974 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_query osd_alive(want up_thru 601 have 601) v1 from osd.2 10.252.24.30:6800/1508 -20822> 2013-04-03 10:34:43.958981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_alive want up_thru 601 from osd.2 10.252.24.30:6800/1508 -20821> 2013-04-03 10:34:43.958985 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_update osd_alive(want up_thru 601 have 601) v1 from osd.2 10.252.24.30:6800/1508 -20820> 2013-04-03 10:34:43.958988 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_alive want up_thru 601 have 601 from osd.2 10.252.24.30:6800/1508 -20819> 2013-04-03 10:34:43.958991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 should_propose -20818> 2013-04-03 10:34:43.959001 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113020 ==== forward(osd_alive(want up_thru 601 have 601) v1) to leader v1 ==== 294+0+0 (2986698178 0 0) 0x273a840 con 0x208a840 -20817> 2013-04-03 10:34:43.959008 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -20816> 2013-04-03 10:34:43.959013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2294700 from 10.252.24.88:6789/0 -20815> 2013-04-03 10:34:43.959033 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_query osd_alive(want up_thru 601 have 601) v1 from osd.3 10.252.24.88:6801/17538 -20814> 2013-04-03 10:34:43.959040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_alive want up_thru 601 from osd.3 10.252.24.88:6801/17538 -20813> 2013-04-03 10:34:43.959043 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_update osd_alive(want up_thru 601 have 601) v1 from osd.3 10.252.24.88:6801/17538 -20812> 2013-04-03 10:34:43.959047 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_alive want up_thru 601 have 601 from osd.3 10.252.24.88:6801/17538 -20811> 2013-04-03 10:34:43.959050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 should_propose -20810> 2013-04-03 10:34:43.992284 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113021 ==== forward(osd_alive(want up_thru 601 have 601) v1) to leader v1 ==== 294+0+0 (4080456625 0 0) 0x273bb80 con 0x208a840 -20809> 2013-04-03 10:34:43.992304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -20808> 2013-04-03 10:34:43.992313 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2294a80 from 10.252.24.88:6789/0 -20807> 2013-04-03 10:34:43.992351 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_query osd_alive(want up_thru 601 have 601) v1 from osd.0 10.252.58.31:6802/2682 -20806> 2013-04-03 10:34:43.992358 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 preprocess_alive want up_thru 601 from osd.0 10.252.58.31:6802/2682 -20805> 2013-04-03 10:34:43.992362 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_update osd_alive(want up_thru 601 have 601) v1 from osd.0 10.252.58.31:6802/2682 -20804> 2013-04-03 10:34:43.992366 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 prepare_alive want up_thru 601 have 601 from osd.0 10.252.58.31:6802/2682 -20803> 2013-04-03 10:34:43.992370 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 should_propose -20802> 2013-04-03 10:34:44.871381 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393985 encode_pending v393986 -20801> 2013-04-03 10:34:44.871412 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427242) queue_proposal bl 876 bytes; ctx = 0x1b91690 -20800> 2013-04-03 10:34:44.871419 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427242) propose_queued 427243 876 bytes -20799> 2013-04-03 10:34:44.872372 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427242 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc780 -20798> 2013-04-03 10:34:44.917284 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113022 ==== paxos(accept lc 427242 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1624622326 0 0) 0x26dc780 con 0x208a840 -20797> 2013-04-03 10:34:45.021153 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427243 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2715500 -20796> 2013-04-03 10:34:45.021192 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427243 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2787000 -20795> 2013-04-03 10:34:45.021251 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos -20794> 2013-04-03 10:34:45.021288 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos version 393986 summary v 393985 -20793> 2013-04-03 10:34:45.021303 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos latest full 393798 -20792> 2013-04-03 10:34:45.021332 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos applying incremental log 393986 2013-04-03 10:34:42.690351 mon.0 10.252.24.25:6789/0 38 : [INF] pgmap v28979: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 4526B/s wr, 1op/s -20791> 2013-04-03 10:34:45.021359 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos applying incremental log 393986 2013-04-03 10:34:43.873213 mon.0 10.252.24.25:6789/0 39 : [INF] osd.2 10.252.24.30:6800/1508 boot -20790> 2013-04-03 10:34:45.021379 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos applying incremental log 393986 2013-04-03 10:34:43.873634 mon.0 10.252.24.25:6789/0 40 : [INF] osdmap e601: 5 osds: 5 up, 5 in -20789> 2013-04-03 10:34:45.021416 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 check_subs -20788> 2013-04-03 10:34:45.021445 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 create_pending v 393987 -20787> 2013-04-03 10:34:45.021467 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 _updated_log for mon.0 10.252.24.25:6789/0 -20786> 2013-04-03 10:34:45.021482 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 39) v1 -- ?+0 0x2224540 con 0x1bac160 -20785> 2013-04-03 10:34:45.021490 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 _updated_log for mon.0 10.252.24.25:6789/0 -20784> 2013-04-03 10:34:45.021503 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 40) v1 -- ?+0 0x2224c40 con 0x1bac160 -20783> 2013-04-03 10:34:45.021553 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos -20782> 2013-04-03 10:34:45.021569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos version 393986 summary v 393986 -20781> 2013-04-03 10:34:45.021586 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20780> 2013-04-03 10:34:45.021609 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 39) v1 ==== 0+0+0 (0 0 0) 0x2224540 con 0x1bac160 -20779> 2013-04-03 10:34:45.021650 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 min_last_epoch_clean 593 -20778> 2013-04-03 10:34:45.021706 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 min_last_epoch_clean 593 -20777> 2013-04-03 10:34:45.021757 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 encode_pending e 602 -20776> 2013-04-03 10:34:45.021793 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427243) queue_proposal bl 317 bytes; ctx = 0x1b91860 -20775> 2013-04-03 10:34:45.021802 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427243) propose_queued 427244 317 bytes -20774> 2013-04-03 10:34:45.023262 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427243 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc280 -20773> 2013-04-03 10:34:45.023301 7fcdffcdb700 10 handle_log_ack log(last 39) v1 -20772> 2013-04-03 10:34:45.023308 7fcdffcdb700 10 logged 2013-04-03 10:34:42.690351 mon.0 10.252.24.25:6789/0 38 : [INF] pgmap v28979: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail; 4526B/s wr, 1op/s -20771> 2013-04-03 10:34:45.023316 7fcdffcdb700 10 logged 2013-04-03 10:34:43.873213 mon.0 10.252.24.25:6789/0 39 : [INF] osd.2 10.252.24.30:6800/1508 boot -20770> 2013-04-03 10:34:45.023323 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 40) v1 ==== 0+0+0 (0 0 0) 0x2224c40 con 0x1bac160 -20769> 2013-04-03 10:34:45.023330 7fcdffcdb700 10 handle_log_ack log(last 40) v1 -20768> 2013-04-03 10:34:45.023331 7fcdffcdb700 10 logged 2013-04-03 10:34:43.873634 mon.0 10.252.24.25:6789/0 40 : [INF] osdmap e601: 5 osds: 5 up, 5 in -20767> 2013-04-03 10:34:45.050727 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113023 ==== paxos(lease_ack lc 427243 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (1982285931 0 0) 0x26dc280 con 0x208a840 -20766> 2013-04-03 10:34:45.050830 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113024 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 19 v187) v2) to leader v1 ==== 560+0+0 (3545716205 0 0) 0x20e48c0 con 0x208a840 -20765> 2013-04-03 10:34:45.050839 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -20764> 2013-04-03 10:34:45.050848 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20e3840 from 10.252.24.88:6789/0 -20763> 2013-04-03 10:34:45.050881 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 19 v187) v2 from mds.0 10.252.24.25:6802/3144 -20762> 2013-04-03 10:34:45.050895 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 19 v187) v2 tid 17425) v2 -- ?+0 0x24f8b40 con 0x208a840 -20761> 2013-04-03 10:34:45.062591 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113025 ==== paxos(accept lc 427243 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2586109126 0 0) 0x2787000 con 0x208a840 -20760> 2013-04-03 10:34:45.063151 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427244 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc280 -20759> 2013-04-03 10:34:45.063183 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427244 fc 427223 pn 0 opn 0) v3 -- ?+0 0x26dc780 -20758> 2013-04-03 10:34:45.063249 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e601 update_from_paxos applying incremental 602 -20757> 2013-04-03 10:34:45.063293 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in -20756> 2013-04-03 10:34:45.064402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 check_osd_map applying osdmap e602 to pg_map -20755> 2013-04-03 10:34:45.064425 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 register_new_pgs checking pg pools for osdmap epoch 602, last_pg_scan 341 -20754> 2013-04-03 10:34:45.064429 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 -20753> 2013-04-03 10:34:45.064434 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -20752> 2013-04-03 10:34:45.064437 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -20751> 2013-04-03 10:34:45.064439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 -20750> 2013-04-03 10:34:45.064442 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 -20749> 2013-04-03 10:34:45.064445 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 -20748> 2013-04-03 10:34:45.064448 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 -20747> 2013-04-03 10:34:45.064451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 -20746> 2013-04-03 10:34:45.064453 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 -20745> 2013-04-03 10:34:45.064464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs -20744> 2013-04-03 10:34:45.064466 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 check_down_pgs -20743> 2013-04-03 10:34:45.064525 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 encode_pending v 28981 -20742> 2013-04-03 10:34:45.064925 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427244) queue_proposal bl 215 bytes; ctx = 0x1b91970 -20741> 2013-04-03 10:34:45.064935 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427244) propose_queued 427245 215 bytes -20740> 2013-04-03 10:34:45.065148 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427244 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2799a00 -20739> 2013-04-03 10:34:45.065181 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 send_pg_creates to 0 pgs -20738> 2013-04-03 10:34:45.065193 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_to_waiting 602 -20737> 2013-04-03 10:34:45.065197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 share_map_with_random_osd no up osd on our session map -20736> 2013-04-03 10:34:45.065199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 update_logger -20735> 2013-04-03 10:34:45.065222 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 create_pending e 603 -20734> 2013-04-03 10:34:45.065226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 remove_redundant_pg_temp -20733> 2013-04-03 10:34:45.065229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 remove_down_pg_temp -20732> 2013-04-03 10:34:45.065247 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 _reply_map 601 from osd.1 10.252.24.25:6805/3572 -20731> 2013-04-03 10:34:45.065267 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_latest to osd.1 10.252.24.25:6805/3572 start 601 -20730> 2013-04-03 10:34:45.065272 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_incremental [601..602] to osd.1 10.252.24.25:6805/3572 -20729> 2013-04-03 10:34:45.065284 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 build_incremental [601..602] -20728> 2013-04-03 10:34:45.065318 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(601..602 src has 101..602) v3 tid 17421) v2 -- ?+0 0x24f8d80 con 0x208a840 -20727> 2013-04-03 10:34:45.065329 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 _reply_map 601 from osd.2 10.252.24.30:6800/1508 -20726> 2013-04-03 10:34:45.065347 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_latest to osd.2 10.252.24.30:6800/1508 start 601 -20725> 2013-04-03 10:34:45.065352 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_incremental [601..602] to osd.2 10.252.24.30:6800/1508 -20724> 2013-04-03 10:34:45.065363 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 build_incremental [601..602] -20723> 2013-04-03 10:34:45.065399 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(601..602 src has 101..602) v3 tid 17422) v2 -- ?+0 0x24f8900 con 0x208a840 -20722> 2013-04-03 10:34:45.065410 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 _reply_map 601 from osd.3 10.252.24.88:6801/17538 -20721> 2013-04-03 10:34:45.065429 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_latest to osd.3 10.252.24.88:6801/17538 start 601 -20720> 2013-04-03 10:34:45.065434 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_incremental [601..602] to osd.3 10.252.24.88:6801/17538 -20719> 2013-04-03 10:34:45.065445 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 build_incremental [601..602] -20718> 2013-04-03 10:34:45.065476 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(601..602 src has 101..602) v3 tid 17423) v2 -- ?+0 0x24f86c0 con 0x208a840 -20717> 2013-04-03 10:34:45.065485 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 _reply_map 601 from osd.0 10.252.58.31:6802/2682 -20716> 2013-04-03 10:34:45.065503 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_latest to osd.0 10.252.58.31:6802/2682 start 601 -20715> 2013-04-03 10:34:45.065508 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 send_incremental [601..602] to osd.0 10.252.58.31:6802/2682 -20714> 2013-04-03 10:34:45.065519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 build_incremental [601..602] -20713> 2013-04-03 10:34:45.065551 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(601..602 src has 101..602) v3 tid 17424) v2 -- ?+0 0x20ad240 con 0x208a840 -20712> 2013-04-03 10:34:45.065560 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 update_logger -20711> 2013-04-03 10:34:45.065568 7fcdffcdb700 0 log [INF] : osdmap e602: 5 osds: 5 up, 5 in -20710> 2013-04-03 10:34:45.065570 7fcdffcdb700 10 send_log to self -20709> 2013-04-03 10:34:45.065570 7fcdffcdb700 10 log_queue is 2 last_log 42 sent 40 num 2 unsent 2 sending 2 -20708> 2013-04-03 10:34:45.065572 7fcdffcdb700 10 will send 2013-04-03 10:34:43.923257 mon.0 10.252.24.25:6789/0 41 : [INF] pgmap v28980: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail -20707> 2013-04-03 10:34:45.065577 7fcdffcdb700 10 will send 2013-04-03 10:34:45.065569 mon.0 10.252.24.25:6789/0 42 : [INF] osdmap e602: 5 osds: 5 up, 5 in -20706> 2013-04-03 10:34:45.065582 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x24e7c00 -20705> 2013-04-03 10:34:45.065609 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x24e7c00 con 0x1bac160 -20704> 2013-04-03 10:34:45.065634 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos -20703> 2013-04-03 10:34:45.065650 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos version 393986 summary v 393986 -20702> 2013-04-03 10:34:45.065660 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -20701> 2013-04-03 10:34:45.065672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 preprocess_log log(2 entries) v1 from mon.0 -20700> 2013-04-03 10:34:45.065692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -20699> 2013-04-03 10:34:45.065705 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 prepare_log log(2 entries) v1 from mon.0 -20698> 2013-04-03 10:34:45.065716 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 logging 2013-04-03 10:34:43.923257 mon.0 10.252.24.25:6789/0 41 : [INF] pgmap v28980: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail -20697> 2013-04-03 10:34:45.065737 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 logging 2013-04-03 10:34:45.065569 mon.0 10.252.24.25:6789/0 42 : [INF] osdmap e602: 5 osds: 5 up, 5 in -20696> 2013-04-03 10:34:45.075829 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113026 ==== paxos(lease_ack lc 427244 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (3814372466 0 0) 0x2799000 con 0x208a840 -20695> 2013-04-03 10:34:45.112925 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113027 ==== paxos(accept lc 427244 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1634856265 0 0) 0x2799280 con 0x208a840 -20694> 2013-04-03 10:34:45.113268 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427245 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2799000 -20693> 2013-04-03 10:34:45.113288 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427245 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2787000 -20692> 2013-04-03 10:34:45.113337 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28980 update_from_paxos applying incremental 28981 -20691> 2013-04-03 10:34:45.113347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 v28981: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail -20690> 2013-04-03 10:34:45.113366 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 send_pg_creates to 0 pgs -20689> 2013-04-03 10:34:45.113369 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 update_logger -20688> 2013-04-03 10:34:45.113379 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 create_pending v 28982 -20687> 2013-04-03 10:34:45.113388 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 check_osd_map already seen 602 >= 602 -20686> 2013-04-03 10:34:45.113390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 update_logger -20685> 2013-04-03 10:34:45.113403 7fcdffcdb700 0 log [INF] : pgmap v28981: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail -20684> 2013-04-03 10:34:45.113404 7fcdffcdb700 10 send_log to self -20683> 2013-04-03 10:34:45.113405 7fcdffcdb700 10 log_queue is 3 last_log 43 sent 42 num 3 unsent 1 sending 1 -20682> 2013-04-03 10:34:45.113407 7fcdffcdb700 10 will send 2013-04-03 10:34:45.065569 mon.0 10.252.24.25:6789/0 42 : [INF] osdmap e602: 5 osds: 5 up, 5 in -20681> 2013-04-03 10:34:45.113415 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2227400 -20680> 2013-04-03 10:34:45.113464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos -20679> 2013-04-03 10:34:45.113480 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos version 393986 summary v 393986 -20678> 2013-04-03 10:34:45.113497 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20677> 2013-04-03 10:34:45.113531 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2227400 con 0x1bac160 -20676> 2013-04-03 10:34:45.113553 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos -20675> 2013-04-03 10:34:45.113567 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos version 393986 summary v 393986 -20674> 2013-04-03 10:34:45.113575 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -20673> 2013-04-03 10:34:45.113585 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 preprocess_log log(1 entries) v1 from mon.0 -20672> 2013-04-03 10:34:45.113602 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 nothing new -20671> 2013-04-03 10:34:45.151187 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113028 ==== paxos(lease_ack lc 427245 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (393431562 0 0) 0x2787000 con 0x208a840 -20670> 2013-04-03 10:34:45.151232 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos -20669> 2013-04-03 10:34:45.151249 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 update_from_paxos version 393986 summary v 393986 -20668> 2013-04-03 10:34:45.151264 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20667> 2013-04-03 10:34:45.377011 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 16 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21675 v187) v2 ==== 289+0+0 (2541267117 0 0) 0x273a580 con 0x200d420 -20666> 2013-04-03 10:34:45.377079 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21675 v187) v2 from mds.? 10.252.24.88:6800/17213 -20665> 2013-04-03 10:34:45.377093 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21675 v187) v2 -- ?+0 0x20e48c0 con 0x200d420 -20664> 2013-04-03 10:34:46.062790 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393986 encode_pending v393987 -20663> 2013-04-03 10:34:46.062820 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427245) queue_proposal bl 651 bytes; ctx = 0x1b919d0 -20662> 2013-04-03 10:34:46.062827 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427245) propose_queued 427246 651 bytes -20661> 2013-04-03 10:34:46.064314 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427245 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116a00 -20660> 2013-04-03 10:34:46.149705 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113029 ==== paxos(accept lc 427245 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3292690631 0 0) 0x2116a00 con 0x208a840 -20659> 2013-04-03 10:34:46.150015 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427246 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787000 -20658> 2013-04-03 10:34:46.150047 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427246 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2799280 -20657> 2013-04-03 10:34:46.150084 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos -20656> 2013-04-03 10:34:46.150112 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos version 393987 summary v 393986 -20655> 2013-04-03 10:34:46.150127 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos latest full 393798 -20654> 2013-04-03 10:34:46.150145 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos applying incremental log 393987 2013-04-03 10:34:43.923257 mon.0 10.252.24.25:6789/0 41 : [INF] pgmap v28980: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail -20653> 2013-04-03 10:34:46.150171 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos applying incremental log 393987 2013-04-03 10:34:45.065569 mon.0 10.252.24.25:6789/0 42 : [INF] osdmap e602: 5 osds: 5 up, 5 in -20652> 2013-04-03 10:34:46.150208 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 check_subs -20651> 2013-04-03 10:34:46.150237 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 create_pending v 393988 -20650> 2013-04-03 10:34:46.150260 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 _updated_log for mon.0 10.252.24.25:6789/0 -20649> 2013-04-03 10:34:46.150273 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 42) v1 -- ?+0 0x2294a80 con 0x1bac160 -20648> 2013-04-03 10:34:46.150322 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos -20647> 2013-04-03 10:34:46.150338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos version 393987 summary v 393987 -20646> 2013-04-03 10:34:46.150357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20645> 2013-04-03 10:34:46.150377 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 42) v1 ==== 0+0+0 (0 0 0) 0x2294a80 con 0x1bac160 -20644> 2013-04-03 10:34:46.150386 7fcdffcdb700 10 handle_log_ack log(last 42) v1 -20643> 2013-04-03 10:34:46.150390 7fcdffcdb700 10 logged 2013-04-03 10:34:43.923257 mon.0 10.252.24.25:6789/0 41 : [INF] pgmap v28980: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail -20642> 2013-04-03 10:34:46.150399 7fcdffcdb700 10 logged 2013-04-03 10:34:45.065569 mon.0 10.252.24.25:6789/0 42 : [INF] osdmap e602: 5 osds: 5 up, 5 in -20641> 2013-04-03 10:34:46.175106 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113030 ==== paxos(lease_ack lc 427246 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (3448986520 0 0) 0x2799280 con 0x208a840 -20640> 2013-04-03 10:34:46.175148 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos -20639> 2013-04-03 10:34:46.175165 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos version 393987 summary v 393987 -20638> 2013-04-03 10:34:46.175180 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20637> 2013-04-03 10:34:47.577008 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 17 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140305 v187) v2 ==== 289+0+0 (1254159669 0 0) 0x2145b00 con 0x2118420 -20636> 2013-04-03 10:34:47.577077 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140305 v187) v2 from mds.? 10.252.58.31:6800/10908 -20635> 2013-04-03 10:34:47.577090 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140305 v187) v2 -- ?+0 0x273a580 con 0x2118420 -20634> 2013-04-03 10:34:48.324315 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 check_down_pgs -20633> 2013-04-03 10:34:48.324345 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 v28981: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail -20632> 2013-04-03 10:34:48.324374 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -20631> 2013-04-03 10:34:48.324396 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in -20630> 2013-04-03 10:34:48.324433 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -20629> 2013-04-03 10:34:48.324464 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos -20628> 2013-04-03 10:34:48.324482 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos version 393987 summary v 393987 -20627> 2013-04-03 10:34:48.324492 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 log -20626> 2013-04-03 10:34:48.324514 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20625> 2013-04-03 10:34:48.324533 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -20624> 2013-04-03 10:34:48.408274 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113031 ==== forward(pg_stats(0 pgs tid 57 v 602) v1) to leader v1 ==== 368+0+0 (2827202945 0 0) 0x273bb80 con 0x208a840 -20623> 2013-04-03 10:34:48.408292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 -20622> 2013-04-03 10:34:48.408301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20ad240 from 10.252.24.88:6789/0 -20621> 2013-04-03 10:34:48.408337 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 preprocess_query pg_stats(0 pgs tid 57 v 602) v1 from osd.4 10.252.24.32:6800/13473 -20620> 2013-04-03 10:34:48.408345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 prepare_update pg_stats(0 pgs tid 57 v 602) v1 from osd.4 10.252.24.32:6800/13473 -20619> 2013-04-03 10:34:48.408349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 prepare_pg_stats pg_stats(0 pgs tid 57 v 602) v1 from osd.4 -20618> 2013-04-03 10:34:48.408355 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 got osd.4 osd_stat(1051 MB used, 56606 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1052 MB used, 56605 MB avail, 59728 MB total, peers []/[])) -20617> 2013-04-03 10:34:48.458525 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 encode_pending v 28982 -20616> 2013-04-03 10:34:48.458560 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427246) queue_proposal bl 265 bytes; ctx = 0x1b918e0 -20615> 2013-04-03 10:34:48.458569 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427246) propose_queued 427247 265 bytes -20614> 2013-04-03 10:34:48.460407 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427246 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116000 -20613> 2013-04-03 10:34:48.542986 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113032 ==== paxos(accept lc 427246 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1425427805 0 0) 0x2116000 con 0x208a840 -20612> 2013-04-03 10:34:48.543285 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427247 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2799280 -20611> 2013-04-03 10:34:48.543311 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427247 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2116a00 -20610> 2013-04-03 10:34:48.543385 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28981 update_from_paxos applying incremental 28982 -20609> 2013-04-03 10:34:48.543396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 v28982: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5673 MB used, 1067 GB / 1079 GB avail -20608> 2013-04-03 10:34:48.543413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 send_pg_creates to 0 pgs -20607> 2013-04-03 10:34:48.543417 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 update_logger -20606> 2013-04-03 10:34:48.543426 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 create_pending v 28983 -20605> 2013-04-03 10:34:48.543436 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 _updated_stats for osd.4 10.252.24.32:6800/13473 -20604> 2013-04-03 10:34:48.543443 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(0 pgs tid 57) v1 tid 17426) v2 -- ?+0 0x20ad900 con 0x208a840 -20603> 2013-04-03 10:34:48.543451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 check_osd_map already seen 602 >= 602 -20602> 2013-04-03 10:34:48.543455 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 update_logger -20601> 2013-04-03 10:34:48.543467 7fcdffcdb700 0 log [INF] : pgmap v28982: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5673 MB used, 1067 GB / 1079 GB avail -20600> 2013-04-03 10:34:48.543468 7fcdffcdb700 10 send_log to self -20599> 2013-04-03 10:34:48.543469 7fcdffcdb700 10 log_queue is 2 last_log 44 sent 42 num 2 unsent 2 sending 2 -20598> 2013-04-03 10:34:48.543471 7fcdffcdb700 10 will send 2013-04-03 10:34:45.113404 mon.0 10.252.24.25:6789/0 43 : [INF] pgmap v28981: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail -20597> 2013-04-03 10:34:48.543480 7fcdffcdb700 10 will send 2013-04-03 10:34:48.543468 mon.0 10.252.24.25:6789/0 44 : [INF] pgmap v28982: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5673 MB used, 1067 GB / 1079 GB avail -20596> 2013-04-03 10:34:48.543485 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x24e7c00 -20595> 2013-04-03 10:34:48.543530 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos -20594> 2013-04-03 10:34:48.543547 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos version 393987 summary v 393987 -20593> 2013-04-03 10:34:48.543565 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20592> 2013-04-03 10:34:48.543583 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x24e7c00 con 0x1bac160 -20591> 2013-04-03 10:34:48.543608 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos -20590> 2013-04-03 10:34:48.543623 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos version 393987 summary v 393987 -20589> 2013-04-03 10:34:48.543632 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -20588> 2013-04-03 10:34:48.543645 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 preprocess_log log(2 entries) v1 from mon.0 -20587> 2013-04-03 10:34:48.543669 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -20586> 2013-04-03 10:34:48.543682 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 prepare_log log(2 entries) v1 from mon.0 -20585> 2013-04-03 10:34:48.543692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 logging 2013-04-03 10:34:45.113404 mon.0 10.252.24.25:6789/0 43 : [INF] pgmap v28981: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail -20584> 2013-04-03 10:34:48.543710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 logging 2013-04-03 10:34:48.543468 mon.0 10.252.24.25:6789/0 44 : [INF] pgmap v28982: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5673 MB used, 1067 GB / 1079 GB avail -20583> 2013-04-03 10:34:48.601515 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113033 ==== paxos(lease_ack lc 427247 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (1227705753 0 0) 0x2116a00 con 0x208a840 -20582> 2013-04-03 10:34:48.601555 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos -20581> 2013-04-03 10:34:48.601571 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 update_from_paxos version 393987 summary v 393987 -20580> 2013-04-03 10:34:48.601586 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20579> 2013-04-03 10:34:48.604258 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113034 ==== forward(pg_stats(71 pgs tid 1 v 602) v1) to leader v1 ==== 26867+0+0 (2721224648 0 0) 0x20e3b00 con 0x208a840 -20578> 2013-04-03 10:34:48.604268 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 -20577> 2013-04-03 10:34:48.604275 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20ad900 from 10.252.24.88:6789/0 -20576> 2013-04-03 10:34:48.604655 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 preprocess_query pg_stats(71 pgs tid 1 v 602) v1 from osd.2 10.252.24.30:6800/1508 -20575> 2013-04-03 10:34:48.604667 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 prepare_update pg_stats(71 pgs tid 1 v 602) v1 from osd.2 10.252.24.30:6800/1508 -20574> 2013-04-03 10:34:48.604672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 prepare_pg_stats pg_stats(71 pgs tid 1 v 602) v1 from osd.2 -20573> 2013-04-03 10:34:48.604677 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,3,4]/[])) -20572> 2013-04-03 10:34:48.922817 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113035 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 20 v187) v2) to leader v1 ==== 560+0+0 (1971510656 0 0) 0x20e4080 con 0x208a840 -20571> 2013-04-03 10:34:48.922831 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -20570> 2013-04-03 10:34:48.922839 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20e4b80 from 10.252.24.88:6789/0 -20569> 2013-04-03 10:34:48.922872 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 20 v187) v2 from mds.0 10.252.24.25:6802/3144 -20568> 2013-04-03 10:34:48.922883 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 20 v187) v2 tid 17428) v2 -- ?+0 0x20ad240 con 0x208a840 -20567> 2013-04-03 10:34:49.377102 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 17 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21676 v187) v2 ==== 289+0+0 (2138150110 0 0) 0x273a2c0 con 0x200d420 -20566> 2013-04-03 10:34:49.377177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21676 v187) v2 from mds.? 10.252.24.88:6800/17213 -20565> 2013-04-03 10:34:49.377191 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21676 v187) v2 -- ?+0 0x20e4080 con 0x200d420 -20564> 2013-04-03 10:34:49.543169 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393987 encode_pending v393988 -20563> 2013-04-03 10:34:49.543197 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427247) queue_proposal bl 734 bytes; ctx = 0x1b91960 -20562> 2013-04-03 10:34:49.543205 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427247) propose_queued 427248 734 bytes -20561> 2013-04-03 10:34:49.544290 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427247 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116500 -20560> 2013-04-03 10:34:49.544394 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 encode_pending v 28983 -20559> 2013-04-03 10:34:49.544623 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427223..427247) queue_proposal bl 26764 bytes; ctx = 0x1b91940 -20558> 2013-04-03 10:34:49.544631 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427223..427247) propose_new_value not active; proposal queued -20557> 2013-04-03 10:34:49.783834 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113036 ==== paxos(accept lc 427247 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (390885083 0 0) 0x2116500 con 0x208a840 -20556> 2013-04-03 10:34:49.784125 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427248 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116a00 -20555> 2013-04-03 10:34:49.784155 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427248 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2116000 -20554> 2013-04-03 10:34:49.784199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos -20553> 2013-04-03 10:34:49.784226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos version 393988 summary v 393987 -20552> 2013-04-03 10:34:49.784241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos latest full 393798 -20551> 2013-04-03 10:34:49.784260 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos applying incremental log 393988 2013-04-03 10:34:45.113404 mon.0 10.252.24.25:6789/0 43 : [INF] pgmap v28981: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail -20550> 2013-04-03 10:34:49.784286 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos applying incremental log 393988 2013-04-03 10:34:48.543468 mon.0 10.252.24.25:6789/0 44 : [INF] pgmap v28982: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5673 MB used, 1067 GB / 1079 GB avail -20549> 2013-04-03 10:34:49.784325 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 check_subs -20548> 2013-04-03 10:34:49.784354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 create_pending v 393989 -20547> 2013-04-03 10:34:49.784377 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 _updated_log for mon.0 10.252.24.25:6789/0 -20546> 2013-04-03 10:34:49.784394 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 44) v1 -- ?+0 0x2294380 con 0x1bac160 -20545> 2013-04-03 10:34:49.784419 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427248) propose_queued 427249 26764 bytes -20544> 2013-04-03 10:34:49.785832 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427248 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2799a00 -20543> 2013-04-03 10:34:49.785860 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 44) v1 ==== 0+0+0 (0 0 0) 0x2294380 con 0x1bac160 -20542> 2013-04-03 10:34:49.785868 7fcdffcdb700 10 handle_log_ack log(last 44) v1 -20541> 2013-04-03 10:34:49.785869 7fcdffcdb700 10 logged 2013-04-03 10:34:45.113404 mon.0 10.252.24.25:6789/0 43 : [INF] pgmap v28981: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5674 MB used, 1067 GB / 1079 GB avail -20540> 2013-04-03 10:34:49.785879 7fcdffcdb700 10 logged 2013-04-03 10:34:48.543468 mon.0 10.252.24.25:6789/0 44 : [INF] pgmap v28982: 240 pgs: 169 active+clean, 71 stale+active+clean; 252 MB data, 5673 MB used, 1067 GB / 1079 GB avail -20539> 2013-04-03 10:34:49.839349 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113037 ==== paxos(lease_ack lc 427248 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (2162440394 0 0) 0x2799a00 con 0x208a840 -20538> 2013-04-03 10:34:49.841355 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113038 ==== forward(pg_stats(45 pgs tid 9 v 602) v1) to leader v1 ==== 17169+0+0 (41906592 0 0) 0x20e3b00 con 0x208a840 -20537> 2013-04-03 10:34:49.841367 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -20536> 2013-04-03 10:34:49.841376 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20ad240 from 10.252.24.88:6789/0 -20535> 2013-04-03 10:34:49.843771 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113039 ==== forward(pg_stats(73 pgs tid 30435 v 602) v1) to leader v1 ==== 27613+0+0 (456466646 0 0) 0x20e4600 con 0x208a840 -20534> 2013-04-03 10:34:49.843780 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -20533> 2013-04-03 10:34:49.843787 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -20532> 2013-04-03 10:34:49.934146 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113040 ==== paxos(accept lc 427248 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1800896100 0 0) 0x2116000 con 0x208a840 -20531> 2013-04-03 10:34:49.935428 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427249 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2799a00 -20530> 2013-04-03 10:34:49.935449 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427249 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2116500 -20529> 2013-04-03 10:34:49.935566 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28982 update_from_paxos applying incremental 28983 -20528> 2013-04-03 10:34:49.935756 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 v28983: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail -20527> 2013-04-03 10:34:49.935787 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 send_pg_creates to 0 pgs -20526> 2013-04-03 10:34:49.935792 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 update_logger -20525> 2013-04-03 10:34:49.935816 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 create_pending v 28984 -20524> 2013-04-03 10:34:49.935827 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 _updated_stats for osd.2 10.252.24.30:6800/1508 -20523> 2013-04-03 10:34:49.935835 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(71 pgs tid 1) v1 tid 17427) v2 -- ?+0 0x20adb40 con 0x208a840 -20522> 2013-04-03 10:34:49.935877 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 preprocess_query pg_stats(45 pgs tid 9 v 602) v1 from osd.1 10.252.24.25:6805/3572 -20521> 2013-04-03 10:34:49.935886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 prepare_update pg_stats(45 pgs tid 9 v 602) v1 from osd.1 10.252.24.25:6805/3572 -20520> 2013-04-03 10:34:49.935891 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 prepare_pg_stats pg_stats(45 pgs tid 9 v 602) v1 from osd.1 -20519> 2013-04-03 10:34:49.935897 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1109 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -20518> 2013-04-03 10:34:49.935972 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 preprocess_query pg_stats(73 pgs tid 30435 v 602) v1 from osd.3 10.252.24.88:6801/17538 -20517> 2013-04-03 10:34:49.935981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 prepare_update pg_stats(73 pgs tid 30435 v 602) v1 from osd.3 10.252.24.88:6801/17538 -20516> 2013-04-03 10:34:49.935985 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 prepare_pg_stats pg_stats(73 pgs tid 30435 v 602) v1 from osd.3 -20515> 2013-04-03 10:34:49.935989 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 got osd.3 osd_stat(1190 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1192 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -20514> 2013-04-03 10:34:49.936076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 check_osd_map already seen 602 >= 602 -20513> 2013-04-03 10:34:49.936082 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 update_logger -20512> 2013-04-03 10:34:49.936093 7fcdffcdb700 0 log [INF] : pgmap v28983: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail -20511> 2013-04-03 10:34:49.936094 7fcdffcdb700 10 send_log to self -20510> 2013-04-03 10:34:49.936095 7fcdffcdb700 10 log_queue is 1 last_log 45 sent 44 num 1 unsent 1 sending 1 -20509> 2013-04-03 10:34:49.936099 7fcdffcdb700 10 will send 2013-04-03 10:34:49.936094 mon.0 10.252.24.25:6789/0 45 : [INF] pgmap v28983: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail -20508> 2013-04-03 10:34:49.936110 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2169e00 -20507> 2013-04-03 10:34:49.936161 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos -20506> 2013-04-03 10:34:49.936180 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos version 393988 summary v 393988 -20505> 2013-04-03 10:34:49.936199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20504> 2013-04-03 10:34:49.936222 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2169e00 con 0x1bac160 -20503> 2013-04-03 10:34:49.936245 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos -20502> 2013-04-03 10:34:49.936260 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos version 393988 summary v 393988 -20501> 2013-04-03 10:34:49.936269 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -20500> 2013-04-03 10:34:49.936284 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 preprocess_log log(1 entries) v1 from mon.0 -20499> 2013-04-03 10:34:49.936303 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -20498> 2013-04-03 10:34:49.936316 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 prepare_log log(1 entries) v1 from mon.0 -20497> 2013-04-03 10:34:49.936326 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 logging 2013-04-03 10:34:49.936094 mon.0 10.252.24.25:6789/0 45 : [INF] pgmap v28983: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail -20496> 2013-04-03 10:34:49.967714 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113041 ==== paxos(lease_ack lc 427249 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (3992935849 0 0) 0x2116500 con 0x208a840 -20495> 2013-04-03 10:34:49.967803 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos -20494> 2013-04-03 10:34:49.967823 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 update_from_paxos version 393988 summary v 393988 -20493> 2013-04-03 10:34:49.967840 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20492> 2013-04-03 10:34:49.969851 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113042 ==== forward(pg_stats(51 pgs tid 406 v 602) v1) to leader v1 ==== 19407+0+0 (2205851591 0 0) 0x1bae600 con 0x208a840 -20491> 2013-04-03 10:34:49.969869 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -20490> 2013-04-03 10:34:49.969879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -20489> 2013-04-03 10:34:49.969920 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 preprocess_query pg_stats(51 pgs tid 406 v 602) v1 from osd.0 10.252.58.31:6802/2682 -20488> 2013-04-03 10:34:49.969929 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 prepare_update pg_stats(51 pgs tid 406 v 602) v1 from osd.0 10.252.58.31:6802/2682 -20487> 2013-04-03 10:34:49.969933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 prepare_pg_stats pg_stats(51 pgs tid 406 v 602) v1 from osd.0 -20486> 2013-04-03 10:34:49.969940 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 got osd.0 osd_stat(1138 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -20485> 2013-04-03 10:34:50.934363 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393988 encode_pending v393989 -20484> 2013-04-03 10:34:50.934402 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427249) queue_proposal bl 424 bytes; ctx = 0x1b91930 -20483> 2013-04-03 10:34:50.934410 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427249) propose_queued 427250 424 bytes -20482> 2013-04-03 10:34:50.935355 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427249 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116280 -20481> 2013-04-03 10:34:50.935452 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 encode_pending v 28984 -20480> 2013-04-03 10:34:50.935939 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427223..427249) queue_proposal bl 63450 bytes; ctx = 0x1b919c0 -20479> 2013-04-03 10:34:50.935948 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427223..427249) propose_new_value not active; proposal queued -20478> 2013-04-03 10:34:50.966845 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113043 ==== paxos(accept lc 427249 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2275133091 0 0) 0x2116280 con 0x208a840 -20477> 2013-04-03 10:34:50.967156 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427250 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116500 -20476> 2013-04-03 10:34:50.967191 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427250 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2116000 -20475> 2013-04-03 10:34:50.967231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos -20474> 2013-04-03 10:34:50.967259 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos version 393989 summary v 393988 -20473> 2013-04-03 10:34:50.967274 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos latest full 393798 -20472> 2013-04-03 10:34:50.967293 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos applying incremental log 393989 2013-04-03 10:34:49.936094 mon.0 10.252.24.25:6789/0 45 : [INF] pgmap v28983: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail -20471> 2013-04-03 10:34:50.967339 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 check_subs -20470> 2013-04-03 10:34:50.967370 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 create_pending v 393990 -20469> 2013-04-03 10:34:50.967393 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 _updated_log for mon.0 10.252.24.25:6789/0 -20468> 2013-04-03 10:34:50.967406 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 45) v1 -- ?+0 0x2294e00 con 0x1bac160 -20467> 2013-04-03 10:34:50.967431 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427250) propose_queued 427251 63450 bytes -20466> 2013-04-03 10:34:50.969150 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427250 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2799c80 -20465> 2013-04-03 10:34:50.969171 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 45) v1 ==== 0+0+0 (0 0 0) 0x2294e00 con 0x1bac160 -20464> 2013-04-03 10:34:50.969179 7fcdffcdb700 10 handle_log_ack log(last 45) v1 -20463> 2013-04-03 10:34:50.969183 7fcdffcdb700 10 logged 2013-04-03 10:34:49.936094 mon.0 10.252.24.25:6789/0 45 : [INF] pgmap v28983: 240 pgs: 240 active+clean; 252 MB data, 5676 MB used, 1067 GB / 1079 GB avail -20462> 2013-04-03 10:34:50.978751 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113044 ==== paxos(lease_ack lc 427250 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (2593323312 0 0) 0x2799c80 con 0x208a840 -20461> 2013-04-03 10:34:51.004133 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113045 ==== paxos(accept lc 427250 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2823850685 0 0) 0x2116000 con 0x208a840 -20460> 2013-04-03 10:34:51.005837 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427251 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2799c80 -20459> 2013-04-03 10:34:51.005856 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427251 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2116280 -20458> 2013-04-03 10:34:51.005965 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28983 update_from_paxos applying incremental 28984 -20457> 2013-04-03 10:34:51.006364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 v28984: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 4547B/s wr, 2op/s -20456> 2013-04-03 10:34:51.006402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 send_pg_creates to 0 pgs -20455> 2013-04-03 10:34:51.006407 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 update_logger -20454> 2013-04-03 10:34:51.006431 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 create_pending v 28985 -20453> 2013-04-03 10:34:51.006443 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 _updated_stats for osd.1 10.252.24.25:6805/3572 -20452> 2013-04-03 10:34:51.006451 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(45 pgs tid 9) v1 tid 17429) v2 -- ?+0 0x20ad900 con 0x208a840 -20451> 2013-04-03 10:34:51.006468 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 _updated_stats for osd.3 10.252.24.88:6801/17538 -20450> 2013-04-03 10:34:51.006473 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(73 pgs tid 30435) v1 tid 17430) v2 -- ?+0 0x20ad240 con 0x208a840 -20449> 2013-04-03 10:34:51.006511 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 _updated_stats for osd.0 10.252.58.31:6802/2682 -20448> 2013-04-03 10:34:51.006517 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(51 pgs tid 406) v1 tid 17431) v2 -- ?+0 0x24f8900 con 0x208a840 -20447> 2013-04-03 10:34:51.007660 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 check_osd_map already seen 602 >= 602 -20446> 2013-04-03 10:34:51.007672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 update_logger -20445> 2013-04-03 10:34:51.007696 7fcdffcdb700 0 log [INF] : pgmap v28984: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 4547B/s wr, 2op/s -20444> 2013-04-03 10:34:51.007705 7fcdffcdb700 10 send_log to self -20443> 2013-04-03 10:34:51.007706 7fcdffcdb700 10 log_queue is 1 last_log 46 sent 45 num 1 unsent 1 sending 1 -20442> 2013-04-03 10:34:51.007707 7fcdffcdb700 10 will send 2013-04-03 10:34:51.007704 mon.0 10.252.24.25:6789/0 46 : [INF] pgmap v28984: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 4547B/s wr, 2op/s -20441> 2013-04-03 10:34:51.007717 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24ebc00 -20440> 2013-04-03 10:34:51.007963 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos -20439> 2013-04-03 10:34:51.007983 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos version 393989 summary v 393989 -20438> 2013-04-03 10:34:51.008002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20437> 2013-04-03 10:34:51.008023 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24ebc00 con 0x1bac160 -20436> 2013-04-03 10:34:51.008046 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos -20435> 2013-04-03 10:34:51.008061 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos version 393989 summary v 393989 -20434> 2013-04-03 10:34:51.008070 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -20433> 2013-04-03 10:34:51.008082 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 preprocess_log log(1 entries) v1 from mon.0 -20432> 2013-04-03 10:34:51.008098 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -20431> 2013-04-03 10:34:51.008109 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 prepare_log log(1 entries) v1 from mon.0 -20430> 2013-04-03 10:34:51.008120 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 logging 2013-04-03 10:34:51.007704 mon.0 10.252.24.25:6789/0 46 : [INF] pgmap v28984: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 4547B/s wr, 2op/s -20429> 2013-04-03 10:34:51.037537 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113046 ==== paxos(lease_ack lc 427251 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (2064487966 0 0) 0x2116280 con 0x208a840 -20428> 2013-04-03 10:34:51.037578 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos -20427> 2013-04-03 10:34:51.037595 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 update_from_paxos version 393989 summary v 393989 -20426> 2013-04-03 10:34:51.037787 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20425> 2013-04-03 10:34:51.577132 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 18 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140306 v187) v2 ==== 289+0+0 (796079282 0 0) 0x2145dc0 con 0x2118420 -20424> 2013-04-03 10:34:51.577203 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140306 v187) v2 from mds.? 10.252.58.31:6800/10908 -20423> 2013-04-03 10:34:51.577217 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140306 v187) v2 -- ?+0 0x20e3840 con 0x2118420 -20422> 2013-04-03 10:34:52.004296 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393989 encode_pending v393990 -20421> 2013-04-03 10:34:52.004325 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427251) queue_proposal bl 443 bytes; ctx = 0x1b919a0 -20420> 2013-04-03 10:34:52.004333 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427251) propose_queued 427252 443 bytes -20419> 2013-04-03 10:34:52.005295 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427251 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116780 -20418> 2013-04-03 10:34:52.111613 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113047 ==== paxos(accept lc 427251 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (728202378 0 0) 0x2116780 con 0x208a840 -20417> 2013-04-03 10:34:52.112621 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427252 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116280 -20416> 2013-04-03 10:34:52.112653 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427252 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2116000 -20415> 2013-04-03 10:34:52.112696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos -20414> 2013-04-03 10:34:52.112736 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos version 393990 summary v 393989 -20413> 2013-04-03 10:34:52.112751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos latest full 393798 -20412> 2013-04-03 10:34:52.112770 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos applying incremental log 393990 2013-04-03 10:34:51.007704 mon.0 10.252.24.25:6789/0 46 : [INF] pgmap v28984: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 4547B/s wr, 2op/s -20411> 2013-04-03 10:34:52.112818 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 check_subs -20410> 2013-04-03 10:34:52.112848 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 create_pending v 393991 -20409> 2013-04-03 10:34:52.112872 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 _updated_log for mon.0 10.252.24.25:6789/0 -20408> 2013-04-03 10:34:52.112885 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 46) v1 -- ?+0 0x2224540 con 0x1bac160 -20407> 2013-04-03 10:34:52.112934 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos -20406> 2013-04-03 10:34:52.112950 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos version 393990 summary v 393990 -20405> 2013-04-03 10:34:52.112969 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20404> 2013-04-03 10:34:52.112990 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 46) v1 ==== 0+0+0 (0 0 0) 0x2224540 con 0x1bac160 -20403> 2013-04-03 10:34:52.112998 7fcdffcdb700 10 handle_log_ack log(last 46) v1 -20402> 2013-04-03 10:34:52.113002 7fcdffcdb700 10 logged 2013-04-03 10:34:51.007704 mon.0 10.252.24.25:6789/0 46 : [INF] pgmap v28984: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 4547B/s wr, 2op/s -20401> 2013-04-03 10:34:52.123464 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113048 ==== paxos(lease_ack lc 427252 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (2602535301 0 0) 0x2116000 con 0x208a840 -20400> 2013-04-03 10:34:52.123501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos -20399> 2013-04-03 10:34:52.123517 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos version 393990 summary v 393990 -20398> 2013-04-03 10:34:52.123534 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20397> 2013-04-03 10:34:52.922926 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113049 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 21 v187) v2) to leader v1 ==== 560+0+0 (1847115002 0 0) 0x1bae8c0 con 0x208a840 -20396> 2013-04-03 10:34:52.922942 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -20395> 2013-04-03 10:34:52.922952 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273a580 from 10.252.24.88:6789/0 -20394> 2013-04-03 10:34:52.922989 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 21 v187) v2 from mds.0 10.252.24.25:6802/3144 -20393> 2013-04-03 10:34:52.923000 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 21 v187) v2 tid 17432) v2 -- ?+0 0x20adb40 con 0x208a840 -20392> 2013-04-03 10:34:53.324671 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 check_down_pgs -20391> 2013-04-03 10:34:53.324701 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 v28984: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 4547B/s wr, 2op/s -20390> 2013-04-03 10:34:53.324731 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -20389> 2013-04-03 10:34:53.324755 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in -20388> 2013-04-03 10:34:53.324793 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -20387> 2013-04-03 10:34:53.324825 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos -20386> 2013-04-03 10:34:53.324843 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos version 393990 summary v 393990 -20385> 2013-04-03 10:34:53.324853 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 log -20384> 2013-04-03 10:34:53.324873 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20383> 2013-04-03 10:34:53.324891 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -20382> 2013-04-03 10:34:53.377284 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 18 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21677 v187) v2 ==== 289+0+0 (1545546659 0 0) 0x273a000 con 0x200d420 -20381> 2013-04-03 10:34:53.377341 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21677 v187) v2 from mds.? 10.252.24.88:6800/17213 -20380> 2013-04-03 10:34:53.377354 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21677 v187) v2 -- ?+0 0x1bae8c0 con 0x200d420 -20379> 2013-04-03 10:34:54.557329 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113050 ==== forward(pg_stats(1 pgs tid 10 v 602) v1) to leader v1 ==== 757+0+0 (3938622691 0 0) 0x2145dc0 con 0x208a840 -20378> 2013-04-03 10:34:54.557349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -20377> 2013-04-03 10:34:54.557360 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -20376> 2013-04-03 10:34:54.557411 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 preprocess_query pg_stats(1 pgs tid 10 v 602) v1 from osd.1 10.252.24.25:6805/3572 -20375> 2013-04-03 10:34:54.557419 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 prepare_update pg_stats(1 pgs tid 10 v 602) v1 from osd.1 10.252.24.25:6805/3572 -20374> 2013-04-03 10:34:54.557423 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 prepare_pg_stats pg_stats(1 pgs tid 10 v 602) v1 from osd.1 -20373> 2013-04-03 10:34:54.557432 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -20372> 2013-04-03 10:34:54.607612 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 encode_pending v 28985 -20371> 2013-04-03 10:34:54.607651 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427252) queue_proposal bl 654 bytes; ctx = 0x1b91980 -20370> 2013-04-03 10:34:54.607660 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427252) propose_queued 427253 654 bytes -20369> 2013-04-03 10:34:54.609455 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427252 fc 0 pn 17600 opn 0) v3 -- ?+0 0x1f91c80 -20368> 2013-04-03 10:34:54.642548 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113051 ==== paxos(accept lc 427252 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4050655090 0 0) 0x1f91c80 con 0x208a840 -20367> 2013-04-03 10:34:54.648674 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427253 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116000 -20366> 2013-04-03 10:34:54.648703 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427253 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2116780 -20365> 2013-04-03 10:34:54.648772 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28984 update_from_paxos applying incremental 28985 -20364> 2013-04-03 10:34:54.648788 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 v28985: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 2629B/s wr, 1op/s -20363> 2013-04-03 10:34:54.648808 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 send_pg_creates to 0 pgs -20362> 2013-04-03 10:34:54.648813 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 update_logger -20361> 2013-04-03 10:34:54.648823 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 create_pending v 28986 -20360> 2013-04-03 10:34:54.648832 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 _updated_stats for osd.1 10.252.24.25:6805/3572 -20359> 2013-04-03 10:34:54.648840 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 10) v1 tid 17433) v2 -- ?+0 0x24f8900 con 0x208a840 -20358> 2013-04-03 10:34:54.648849 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 check_osd_map already seen 602 >= 602 -20357> 2013-04-03 10:34:54.648852 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 update_logger -20356> 2013-04-03 10:34:54.648863 7fcdffcdb700 0 log [INF] : pgmap v28985: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 2629B/s wr, 1op/s -20355> 2013-04-03 10:34:54.648865 7fcdffcdb700 10 send_log to self -20354> 2013-04-03 10:34:54.648865 7fcdffcdb700 10 log_queue is 1 last_log 47 sent 46 num 1 unsent 1 sending 1 -20353> 2013-04-03 10:34:54.648867 7fcdffcdb700 10 will send 2013-04-03 10:34:54.648864 mon.0 10.252.24.25:6789/0 47 : [INF] pgmap v28985: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 2629B/s wr, 1op/s -20352> 2013-04-03 10:34:54.648873 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24ebc00 -20351> 2013-04-03 10:34:54.648921 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos -20350> 2013-04-03 10:34:54.648942 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos version 393990 summary v 393990 -20349> 2013-04-03 10:34:54.648961 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20348> 2013-04-03 10:34:54.648981 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24ebc00 con 0x1bac160 -20347> 2013-04-03 10:34:54.649003 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos -20346> 2013-04-03 10:34:54.649018 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos version 393990 summary v 393990 -20345> 2013-04-03 10:34:54.649028 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -20344> 2013-04-03 10:34:54.649040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 preprocess_log log(1 entries) v1 from mon.0 -20343> 2013-04-03 10:34:54.649059 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -20342> 2013-04-03 10:34:54.649072 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 prepare_log log(1 entries) v1 from mon.0 -20341> 2013-04-03 10:34:54.649084 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 logging 2013-04-03 10:34:54.648864 mon.0 10.252.24.25:6789/0 47 : [INF] pgmap v28985: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 2629B/s wr, 1op/s -20340> 2013-04-03 10:34:54.676932 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113052 ==== paxos(lease_ack lc 427253 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (793672382 0 0) 0x2116780 con 0x208a840 -20339> 2013-04-03 10:34:54.676970 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos -20338> 2013-04-03 10:34:54.676987 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 update_from_paxos version 393990 summary v 393990 -20337> 2013-04-03 10:34:54.677002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20336> 2013-04-03 10:34:55.577260 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 19 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140307 v187) v2 ==== 289+0+0 (203262927 0 0) 0x21452c0 con 0x2118420 -20335> 2013-04-03 10:34:55.577339 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140307 v187) v2 from mds.? 10.252.58.31:6800/10908 -20334> 2013-04-03 10:34:55.577353 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140307 v187) v2 -- ?+0 0x2145dc0 con 0x2118420 -20333> 2013-04-03 10:34:55.642731 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393990 encode_pending v393991 -20332> 2013-04-03 10:34:55.642758 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427253) queue_proposal bl 443 bytes; ctx = 0x1b91950 -20331> 2013-04-03 10:34:55.642765 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427253) propose_queued 427254 443 bytes -20330> 2013-04-03 10:34:55.644041 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427253 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4780 -20329> 2013-04-03 10:34:55.669698 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113053 ==== paxos(accept lc 427253 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4269913489 0 0) 0x20f4780 con 0x208a840 -20328> 2013-04-03 10:34:55.669988 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427254 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116780 -20327> 2013-04-03 10:34:55.670019 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427254 fc 427223 pn 0 opn 0) v3 -- ?+0 0x1f91c80 -20326> 2013-04-03 10:34:55.670055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos -20325> 2013-04-03 10:34:55.670082 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos version 393991 summary v 393990 -20324> 2013-04-03 10:34:55.670097 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos latest full 393798 -20323> 2013-04-03 10:34:55.670115 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos applying incremental log 393991 2013-04-03 10:34:54.648864 mon.0 10.252.24.25:6789/0 47 : [INF] pgmap v28985: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 2629B/s wr, 1op/s -20322> 2013-04-03 10:34:55.670170 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 check_subs -20321> 2013-04-03 10:34:55.670202 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 create_pending v 393992 -20320> 2013-04-03 10:34:55.670225 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 _updated_log for mon.0 10.252.24.25:6789/0 -20319> 2013-04-03 10:34:55.670238 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 47) v1 -- ?+0 0x22241c0 con 0x1bac160 -20318> 2013-04-03 10:34:55.670286 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos -20317> 2013-04-03 10:34:55.670302 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos version 393991 summary v 393991 -20316> 2013-04-03 10:34:55.670321 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20315> 2013-04-03 10:34:55.670341 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 47) v1 ==== 0+0+0 (0 0 0) 0x22241c0 con 0x1bac160 -20314> 2013-04-03 10:34:55.670349 7fcdffcdb700 10 handle_log_ack log(last 47) v1 -20313> 2013-04-03 10:34:55.670352 7fcdffcdb700 10 logged 2013-04-03 10:34:54.648864 mon.0 10.252.24.25:6789/0 47 : [INF] pgmap v28985: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 2629B/s wr, 1op/s -20312> 2013-04-03 10:34:55.682113 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113054 ==== paxos(lease_ack lc 427254 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (3743587911 0 0) 0x1f91c80 con 0x208a840 -20311> 2013-04-03 10:34:55.682148 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos -20310> 2013-04-03 10:34:55.682164 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos version 393991 summary v 393991 -20309> 2013-04-03 10:34:55.682181 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20308> 2013-04-03 10:34:56.923135 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113055 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 22 v187) v2) to leader v1 ==== 560+0+0 (970364877 0 0) 0x273b080 con 0x208a840 -20307> 2013-04-03 10:34:56.923157 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -20306> 2013-04-03 10:34:56.923168 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273a2c0 from 10.252.24.88:6789/0 -20305> 2013-04-03 10:34:56.923221 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 22 v187) v2 from mds.0 10.252.24.25:6802/3144 -20304> 2013-04-03 10:34:56.923234 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 22 v187) v2 tid 17434) v2 -- ?+0 0x20adb40 con 0x208a840 -20303> 2013-04-03 10:34:57.377282 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 19 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21678 v187) v2 ==== 289+0+0 (967633444 0 0) 0x28d3dc0 con 0x200d420 -20302> 2013-04-03 10:34:57.377360 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21678 v187) v2 from mds.? 10.252.24.88:6800/17213 -20301> 2013-04-03 10:34:57.377374 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21678 v187) v2 -- ?+0 0x273b080 con 0x200d420 -20300> 2013-04-03 10:34:58.325040 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 check_down_pgs -20299> 2013-04-03 10:34:58.325069 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 v28985: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 2629B/s wr, 1op/s -20298> 2013-04-03 10:34:58.325097 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -20297> 2013-04-03 10:34:58.325121 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in -20296> 2013-04-03 10:34:58.325159 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -20295> 2013-04-03 10:34:58.325190 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos -20294> 2013-04-03 10:34:58.325212 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos version 393991 summary v 393991 -20293> 2013-04-03 10:34:58.325222 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 log -20292> 2013-04-03 10:34:58.325239 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20291> 2013-04-03 10:34:58.325258 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -20290> 2013-04-03 10:34:58.477479 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113056 ==== forward(pg_stats(1 pgs tid 2 v 602) v1) to leader v1 ==== 757+0+0 (1965908128 0 0) 0x21452c0 con 0x208a840 -20289> 2013-04-03 10:34:58.477494 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 -20288> 2013-04-03 10:34:58.477504 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -20287> 2013-04-03 10:34:58.477539 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 preprocess_query pg_stats(1 pgs tid 2 v 602) v1 from osd.2 10.252.24.30:6800/1508 -20286> 2013-04-03 10:34:58.477547 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 prepare_update pg_stats(1 pgs tid 2 v 602) v1 from osd.2 10.252.24.30:6800/1508 -20285> 2013-04-03 10:34:58.477551 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 prepare_pg_stats pg_stats(1 pgs tid 2 v 602) v1 from osd.2 -20284> 2013-04-03 10:34:58.477557 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -20283> 2013-04-03 10:34:58.527734 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 encode_pending v 28986 -20282> 2013-04-03 10:34:58.527764 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427254) queue_proposal bl 654 bytes; ctx = 0x1b91890 -20281> 2013-04-03 10:34:58.527773 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427254) propose_queued 427255 654 bytes -20280> 2013-04-03 10:34:58.528752 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427254 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4280 -20279> 2013-04-03 10:34:58.713615 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113057 ==== paxos(accept lc 427254 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4169236381 0 0) 0x20f4280 con 0x208a840 -20278> 2013-04-03 10:34:58.713937 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427255 fc 0 pn 17600 opn 0) v3 -- ?+0 0x1f91c80 -20277> 2013-04-03 10:34:58.713968 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427255 fc 427223 pn 0 opn 0) v3 -- ?+0 0x20f4780 -20276> 2013-04-03 10:34:58.714037 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28985 update_from_paxos applying incremental 28986 -20275> 2013-04-03 10:34:58.714055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 v28986: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s -20274> 2013-04-03 10:34:58.714073 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 send_pg_creates to 0 pgs -20273> 2013-04-03 10:34:58.714078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 update_logger -20272> 2013-04-03 10:34:58.714088 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 create_pending v 28987 -20271> 2013-04-03 10:34:58.714098 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 _updated_stats for osd.2 10.252.24.30:6800/1508 -20270> 2013-04-03 10:34:58.714104 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 2) v1 tid 17435) v2 -- ?+0 0x24f8900 con 0x208a840 -20269> 2013-04-03 10:34:58.714113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 check_osd_map already seen 602 >= 602 -20268> 2013-04-03 10:34:58.714118 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 update_logger -20267> 2013-04-03 10:34:58.714129 7fcdffcdb700 0 log [INF] : pgmap v28986: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s -20266> 2013-04-03 10:34:58.714132 7fcdffcdb700 10 send_log to self -20265> 2013-04-03 10:34:58.714132 7fcdffcdb700 10 log_queue is 1 last_log 48 sent 47 num 1 unsent 1 sending 1 -20264> 2013-04-03 10:34:58.714134 7fcdffcdb700 10 will send 2013-04-03 10:34:58.714131 mon.0 10.252.24.25:6789/0 48 : [INF] pgmap v28986: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s -20263> 2013-04-03 10:34:58.714142 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24ebc00 -20262> 2013-04-03 10:34:58.714187 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos -20261> 2013-04-03 10:34:58.714205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos version 393991 summary v 393991 -20260> 2013-04-03 10:34:58.714223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20259> 2013-04-03 10:34:58.714243 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24ebc00 con 0x1bac160 -20258> 2013-04-03 10:34:58.714264 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos -20257> 2013-04-03 10:34:58.714280 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos version 393991 summary v 393991 -20256> 2013-04-03 10:34:58.714291 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -20255> 2013-04-03 10:34:58.714303 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 preprocess_log log(1 entries) v1 from mon.0 -20254> 2013-04-03 10:34:58.714323 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -20253> 2013-04-03 10:34:58.714336 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 prepare_log log(1 entries) v1 from mon.0 -20252> 2013-04-03 10:34:58.714347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 logging 2013-04-03 10:34:58.714131 mon.0 10.252.24.25:6789/0 48 : [INF] pgmap v28986: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s -20251> 2013-04-03 10:34:58.727259 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113058 ==== paxos(lease_ack lc 427255 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (3254949705 0 0) 0x20f4780 con 0x208a840 -20250> 2013-04-03 10:34:58.727296 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos -20249> 2013-04-03 10:34:58.727313 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 update_from_paxos version 393991 summary v 393991 -20248> 2013-04-03 10:34:58.727328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20247> 2013-04-03 10:34:59.577402 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 20 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140308 v187) v2 ==== 289+0+0 (3826563004 0 0) 0x2145000 con 0x2118420 -20246> 2013-04-03 10:34:59.577474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140308 v187) v2 from mds.? 10.252.58.31:6800/10908 -20245> 2013-04-03 10:34:59.577488 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140308 v187) v2 -- ?+0 0x21452c0 con 0x2118420 -20244> 2013-04-03 10:34:59.713797 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393991 encode_pending v393992 -20243> 2013-04-03 10:34:59.713823 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427255) queue_proposal bl 443 bytes; ctx = 0x1b91900 -20242> 2013-04-03 10:34:59.713832 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427255) propose_queued 427256 443 bytes -20241> 2013-04-03 10:34:59.715071 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427255 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4c80 -20240> 2013-04-03 10:34:59.752342 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113059 ==== paxos(accept lc 427255 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (56042102 0 0) 0x20f4c80 con 0x208a840 -20239> 2013-04-03 10:34:59.752643 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427256 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4780 -20238> 2013-04-03 10:34:59.752675 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427256 fc 427223 pn 0 opn 0) v3 -- ?+0 0x20f4280 -20237> 2013-04-03 10:34:59.752713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos -20236> 2013-04-03 10:34:59.752740 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393991 -20235> 2013-04-03 10:34:59.752754 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos latest full 393798 -20234> 2013-04-03 10:34:59.752773 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos applying incremental log 393992 2013-04-03 10:34:58.714131 mon.0 10.252.24.25:6789/0 48 : [INF] pgmap v28986: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s -20233> 2013-04-03 10:34:59.752821 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 check_subs -20232> 2013-04-03 10:34:59.752851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 create_pending v 393993 -20231> 2013-04-03 10:34:59.752874 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 _updated_log for mon.0 10.252.24.25:6789/0 -20230> 2013-04-03 10:34:59.752887 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 48) v1 -- ?+0 0x22248c0 con 0x1bac160 -20229> 2013-04-03 10:34:59.752935 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos -20228> 2013-04-03 10:34:59.752951 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393992 -20227> 2013-04-03 10:34:59.752969 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20226> 2013-04-03 10:34:59.752991 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 48) v1 ==== 0+0+0 (0 0 0) 0x22248c0 con 0x1bac160 -20225> 2013-04-03 10:34:59.752999 7fcdffcdb700 10 handle_log_ack log(last 48) v1 -20224> 2013-04-03 10:34:59.753000 7fcdffcdb700 10 logged 2013-04-03 10:34:58.714131 mon.0 10.252.24.25:6789/0 48 : [INF] pgmap v28986: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s -20223> 2013-04-03 10:34:59.764738 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113060 ==== paxos(lease_ack lc 427256 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (3698884695 0 0) 0x20f4280 con 0x208a840 -20222> 2013-04-03 10:34:59.764775 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos -20221> 2013-04-03 10:34:59.764792 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393992 -20220> 2013-04-03 10:34:59.764809 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20219> 2013-04-03 10:35:00.153188 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x26dc000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:32963/0 -20218> 2013-04-03 10:35:00.153219 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x26dc000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:32963/0) -20217> 2013-04-03 10:35:00.153232 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x26dc000 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -20216> 2013-04-03 10:35:00.153437 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x26dc000 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -20215> 2013-04-03 10:35:00.923225 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113061 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 23 v187) v2) to leader v1 ==== 560+0+0 (3508453537 0 0) 0x273bb80 con 0x208a840 -20214> 2013-04-03 10:35:00.923247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -20213> 2013-04-03 10:35:00.923258 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20e3dc0 from 10.252.24.88:6789/0 -20212> 2013-04-03 10:35:00.923310 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 23 v187) v2 from mds.0 10.252.24.25:6802/3144 -20211> 2013-04-03 10:35:00.923322 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 23 v187) v2 tid 17436) v2 -- ?+0 0x20adb40 con 0x208a840 -20210> 2013-04-03 10:35:01.377410 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 20 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21679 v187) v2 ==== 289+0+0 (448943449 0 0) 0x28d3b00 con 0x200d420 -20209> 2013-04-03 10:35:01.377487 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21679 v187) v2 from mds.? 10.252.24.88:6800/17213 -20208> 2013-04-03 10:35:01.377502 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21679 v187) v2 -- ?+0 0x273bb80 con 0x200d420 -20207> 2013-04-03 10:35:02.752758 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427256 fc 427223 pn 0 opn 0) v3 -- ?+0 0x20f4a00 -20206> 2013-04-03 10:35:02.753471 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113062 ==== paxos(lease_ack lc 427256 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (3651335859 0 0) 0x20f4a00 con 0x208a840 -20205> 2013-04-03 10:35:02.753543 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos -20204> 2013-04-03 10:35:02.753572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393992 -20203> 2013-04-03 10:35:02.753590 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20202> 2013-04-03 10:35:03.325405 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 check_down_pgs -20201> 2013-04-03 10:35:03.325430 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 v28986: 240 pgs: 240 active+clean; 252 MB data, 5666 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s -20200> 2013-04-03 10:35:03.325457 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -20199> 2013-04-03 10:35:03.325481 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in -20198> 2013-04-03 10:35:03.325519 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -20197> 2013-04-03 10:35:03.325550 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos -20196> 2013-04-03 10:35:03.325571 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393992 -20195> 2013-04-03 10:35:03.325581 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 log -20194> 2013-04-03 10:35:03.325598 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20193> 2013-04-03 10:35:03.325617 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -20192> 2013-04-03 10:35:03.478106 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113063 ==== forward(pg_stats(1 pgs tid 3 v 602) v1) to leader v1 ==== 757+0+0 (4142229478 0 0) 0x2145000 con 0x208a840 -20191> 2013-04-03 10:35:03.478123 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 -20190> 2013-04-03 10:35:03.478133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -20189> 2013-04-03 10:35:03.478169 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 preprocess_query pg_stats(1 pgs tid 3 v 602) v1 from osd.2 10.252.24.30:6800/1508 -20188> 2013-04-03 10:35:03.478177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 prepare_update pg_stats(1 pgs tid 3 v 602) v1 from osd.2 10.252.24.30:6800/1508 -20187> 2013-04-03 10:35:03.478181 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 prepare_pg_stats pg_stats(1 pgs tid 3 v 602) v1 from osd.2 -20186> 2013-04-03 10:35:03.478187 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -20185> 2013-04-03 10:35:03.528367 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 encode_pending v 28987 -20184> 2013-04-03 10:35:03.528398 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427256) queue_proposal bl 654 bytes; ctx = 0x1b91ac0 -20183> 2013-04-03 10:35:03.528407 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427256) propose_queued 427257 654 bytes -20182> 2013-04-03 10:35:03.531398 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427256 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4500 -20181> 2013-04-03 10:35:03.577513 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 21 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140309 v187) v2 ==== 289+0+0 (3346663617 0 0) 0x2145580 con 0x2118420 -20180> 2013-04-03 10:35:03.577569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140309 v187) v2 from mds.? 10.252.58.31:6800/10908 -20179> 2013-04-03 10:35:03.577581 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140309 v187) v2 -- ?+0 0x2145000 con 0x2118420 -20178> 2013-04-03 10:35:03.712034 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113064 ==== paxos(accept lc 427256 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1031050856 0 0) 0x20f4500 con 0x208a840 -20177> 2013-04-03 10:35:03.712332 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427257 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4a00 -20176> 2013-04-03 10:35:03.712360 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427257 fc 427223 pn 0 opn 0) v3 -- ?+0 0x20f4280 -20175> 2013-04-03 10:35:03.712428 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28986 update_from_paxos applying incremental 28987 -20174> 2013-04-03 10:35:03.712444 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28987 v28987: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3214B/s wr, 0op/s -20173> 2013-04-03 10:35:03.712463 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28987 send_pg_creates to 0 pgs -20172> 2013-04-03 10:35:03.712467 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28987 update_logger -20171> 2013-04-03 10:35:03.712476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28987 create_pending v 28988 -20170> 2013-04-03 10:35:03.712486 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28987 _updated_stats for osd.2 10.252.24.30:6800/1508 -20169> 2013-04-03 10:35:03.712493 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 3) v1 tid 17437) v2 -- ?+0 0x24f8900 con 0x208a840 -20168> 2013-04-03 10:35:03.712502 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28987 check_osd_map already seen 602 >= 602 -20167> 2013-04-03 10:35:03.712506 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28987 update_logger -20166> 2013-04-03 10:35:03.712517 7fcdffcdb700 0 log [INF] : pgmap v28987: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3214B/s wr, 0op/s -20165> 2013-04-03 10:35:03.712519 7fcdffcdb700 10 send_log to self -20164> 2013-04-03 10:35:03.712519 7fcdffcdb700 10 log_queue is 1 last_log 49 sent 48 num 1 unsent 1 sending 1 -20163> 2013-04-03 10:35:03.712521 7fcdffcdb700 10 will send 2013-04-03 10:35:03.712518 mon.0 10.252.24.25:6789/0 49 : [INF] pgmap v28987: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3214B/s wr, 0op/s -20162> 2013-04-03 10:35:03.712527 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24ebc00 -20161> 2013-04-03 10:35:03.712575 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos -20160> 2013-04-03 10:35:03.712593 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393992 -20159> 2013-04-03 10:35:03.712610 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20158> 2013-04-03 10:35:03.712630 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24ebc00 con 0x1bac160 -20157> 2013-04-03 10:35:03.712652 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos -20156> 2013-04-03 10:35:03.712667 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393992 -20155> 2013-04-03 10:35:03.712676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -20154> 2013-04-03 10:35:03.712687 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 preprocess_log log(1 entries) v1 from mon.0 -20153> 2013-04-03 10:35:03.712706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -20152> 2013-04-03 10:35:03.712719 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 prepare_log log(1 entries) v1 from mon.0 -20151> 2013-04-03 10:35:03.712729 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 logging 2013-04-03 10:35:03.712518 mon.0 10.252.24.25:6789/0 49 : [INF] pgmap v28987: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3214B/s wr, 0op/s -20150> 2013-04-03 10:35:03.738065 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113065 ==== paxos(lease_ack lc 427257 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (2836676692 0 0) 0x20f4280 con 0x208a840 -20149> 2013-04-03 10:35:03.738108 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos -20148> 2013-04-03 10:35:03.738125 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 update_from_paxos version 393992 summary v 393992 -20147> 2013-04-03 10:35:03.738141 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20146> 2013-04-03 10:35:04.712233 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393992 encode_pending v393993 -20145> 2013-04-03 10:35:04.712262 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427257) queue_proposal bl 443 bytes; ctx = 0x1b91910 -20144> 2013-04-03 10:35:04.712270 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427257) propose_queued 427258 443 bytes -20143> 2013-04-03 10:35:04.713502 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427257 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2799780 -20142> 2013-04-03 10:35:04.751608 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113066 ==== paxos(accept lc 427257 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4074718174 0 0) 0x2799780 con 0x208a840 -20141> 2013-04-03 10:35:04.751906 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427258 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4280 -20140> 2013-04-03 10:35:04.751939 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427258 fc 427223 pn 0 opn 0) v3 -- ?+0 0x20f4500 -20139> 2013-04-03 10:35:04.751978 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos -20138> 2013-04-03 10:35:04.752207 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos version 393993 summary v 393992 -20137> 2013-04-03 10:35:04.752222 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos latest full 393798 -20136> 2013-04-03 10:35:04.752243 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos applying incremental log 393993 2013-04-03 10:35:03.712518 mon.0 10.252.24.25:6789/0 49 : [INF] pgmap v28987: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3214B/s wr, 0op/s -20135> 2013-04-03 10:35:04.752288 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 check_subs -20134> 2013-04-03 10:35:04.752317 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 create_pending v 393994 -20133> 2013-04-03 10:35:04.752340 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 _updated_log for mon.0 10.252.24.25:6789/0 -20132> 2013-04-03 10:35:04.752353 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 49) v1 -- ?+0 0x2224e00 con 0x1bac160 -20131> 2013-04-03 10:35:04.752402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos -20130> 2013-04-03 10:35:04.752418 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos version 393993 summary v 393993 -20129> 2013-04-03 10:35:04.752436 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20128> 2013-04-03 10:35:04.752459 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 49) v1 ==== 0+0+0 (0 0 0) 0x2224e00 con 0x1bac160 -20127> 2013-04-03 10:35:04.752467 7fcdffcdb700 10 handle_log_ack log(last 49) v1 -20126> 2013-04-03 10:35:04.752469 7fcdffcdb700 10 logged 2013-04-03 10:35:03.712518 mon.0 10.252.24.25:6789/0 49 : [INF] pgmap v28987: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3214B/s wr, 0op/s -20125> 2013-04-03 10:35:04.765238 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113067 ==== paxos(lease_ack lc 427258 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (3408012903 0 0) 0x20f4500 con 0x208a840 -20124> 2013-04-03 10:35:04.765275 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos -20123> 2013-04-03 10:35:04.765292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos version 393993 summary v 393993 -20122> 2013-04-03 10:35:04.765308 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20121> 2013-04-03 10:35:04.923339 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113068 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 24 v187) v2) to leader v1 ==== 560+0+0 (3078526409 0 0) 0x20e4080 con 0x208a840 -20120> 2013-04-03 10:35:04.923353 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -20119> 2013-04-03 10:35:04.923364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20e48c0 from 10.252.24.88:6789/0 -20118> 2013-04-03 10:35:04.923396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 24 v187) v2 from mds.0 10.252.24.25:6802/3144 -20117> 2013-04-03 10:35:04.923407 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 24 v187) v2 tid 17438) v2 -- ?+0 0x20adb40 con 0x208a840 -20116> 2013-04-03 10:35:05.377535 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 21 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21680 v187) v2 ==== 289+0+0 (3512860161 0 0) 0x28d3840 con 0x200d420 -20115> 2013-04-03 10:35:05.377612 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21680 v187) v2 from mds.? 10.252.24.88:6800/17213 -20114> 2013-04-03 10:35:05.377626 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21680 v187) v2 -- ?+0 0x20e4080 con 0x200d420 -20113> 2013-04-03 10:35:07.577642 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 22 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140310 v187) v2 ==== 289+0+0 (2731133254 0 0) 0x28d4b80 con 0x2118420 -20112> 2013-04-03 10:35:07.577713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140310 v187) v2 from mds.? 10.252.58.31:6800/10908 -20111> 2013-04-03 10:35:07.577727 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140310 v187) v2 -- ?+0 0x28d3840 con 0x2118420 -20110> 2013-04-03 10:35:07.752009 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427258 fc 427223 pn 0 opn 0) v3 -- ?+0 0x26dc780 -20109> 2013-04-03 10:35:07.752847 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113069 ==== paxos(lease_ack lc 427258 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (1406082563 0 0) 0x26dc780 con 0x208a840 -20108> 2013-04-03 10:35:07.752892 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos -20107> 2013-04-03 10:35:07.752911 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393993 update_from_paxos version 393993 summary v 393993 -20106> 2013-04-03 10:35:07.752928 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -20105> 2013-04-03 10:35:08.157374 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) -20104> 2013-04-03 10:35:08.157417 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader -20103> 2013-04-03 10:35:08.325769 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28987 check_down_pgs -20102> 2013-04-03 10:35:08.325796 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28987 v28987: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 3214B/s wr, 0op/s -20101> 2013-04-03 10:35:08.325823 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -20100> 2013-04-03 10:35:08.325844 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -20099> 2013-04-03 10:35:08.325883 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -20098> 2013-04-03 10:35:08.325913 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos -20097> 2013-04-03 10:35:08.325930 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos version 393993 summary v 393993 -20096> 2013-04-03 10:35:08.325939 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393993 log -20095> 2013-04-03 10:35:08.325955 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -20094> 2013-04-03 10:35:08.325973 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -20093> 2013-04-03 10:35:08.923491 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113070 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 25 v187) v2) to leader v1 ==== 560+0+0 (3191294922 0 0) 0x2145580 con 0x208a840 -20092> 2013-04-03 10:35:08.923506 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -20091> 2013-04-03 10:35:08.923515 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4340 from 10.252.24.88:6789/0 -20090> 2013-04-03 10:35:08.923552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 25 v187) v2 from mds.0 10.252.24.25:6802/3144 -20089> 2013-04-03 10:35:08.923563 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 25 v187) v2 tid 17439) v2 -- ?+0 0x24f8900 con 0x208a840 -20088> 2013-04-03 10:35:09.377642 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 22 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21681 v187) v2 ==== 289+0+0 (4061086076 0 0) 0x28d3580 con 0x200d420 -20087> 2013-04-03 10:35:09.377723 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21681 v187) v2 from mds.? 10.252.24.88:6800/17213 -20086> 2013-04-03 10:35:09.377736 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21681 v187) v2 -- ?+0 0x2145580 con 0x200d420 -20085> 2013-04-03 10:35:09.558938 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113071 ==== forward(pg_stats(1 pgs tid 11 v 602) v1) to leader v1 ==== 757+0+0 (4180705646 0 0) 0x28d4b80 con 0x208a840 -20084> 2013-04-03 10:35:09.558952 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -20083> 2013-04-03 10:35:09.558961 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -20082> 2013-04-03 10:35:09.558994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28987 preprocess_query pg_stats(1 pgs tid 11 v 602) v1 from osd.1 10.252.24.25:6805/3572 -20081> 2013-04-03 10:35:09.559002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28987 prepare_update pg_stats(1 pgs tid 11 v 602) v1 from osd.1 10.252.24.25:6805/3572 -20080> 2013-04-03 10:35:09.559005 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28987 prepare_pg_stats pg_stats(1 pgs tid 11 v 602) v1 from osd.1 -20079> 2013-04-03 10:35:09.559011 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28987 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -20078> 2013-04-03 10:35:09.609198 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28987 encode_pending v 28988 -20077> 2013-04-03 10:35:09.609229 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427223..427258) queue_proposal bl 654 bytes; ctx = 0x1b91920 -20076> 2013-04-03 10:35:09.609238 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427223..427258) propose_queued 427259 654 bytes -20075> 2013-04-03 10:35:09.611094 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427258 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc280 -20074> 2013-04-03 10:35:09.641986 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113072 ==== paxos(accept lc 427258 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (607540595 0 0) 0x26dc280 con 0x208a840 -20073> 2013-04-03 10:35:09.642370 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427259 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc780 -20072> 2013-04-03 10:35:09.642396 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427259 fc 427223 pn 0 opn 0) v3 -- ?+0 0x20f4500 -20071> 2013-04-03 10:35:09.642465 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28987 update_from_paxos applying incremental 28988 -20070> 2013-04-03 10:35:09.642480 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 v28988: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 1848B/s wr, 0op/s -20069> 2013-04-03 10:35:09.642499 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 send_pg_creates to 0 pgs -20068> 2013-04-03 10:35:09.642502 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 update_logger -20067> 2013-04-03 10:35:09.642511 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 create_pending v 28989 -20066> 2013-04-03 10:35:09.642519 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28988 _updated_stats for osd.1 10.252.24.25:6805/3572 -20065> 2013-04-03 10:35:09.642526 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 11) v1 tid 17440) v2 -- ?+0 0x20adb40 con 0x208a840 -20064> 2013-04-03 10:35:09.642537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 check_osd_map already seen 602 >= 602 -20063> 2013-04-03 10:35:09.642540 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 update_logger -20062> 2013-04-03 10:35:09.642550 7fcdffcdb700 0 log [INF] : pgmap v28988: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 1848B/s wr, 0op/s -20061> 2013-04-03 10:35:09.642551 7fcdffcdb700 10 send_log to self -20060> 2013-04-03 10:35:09.642552 7fcdffcdb700 10 log_queue is 1 last_log 50 sent 49 num 1 unsent 1 sending 1 -20059> 2013-04-03 10:35:09.642554 7fcdffcdb700 10 will send 2013-04-03 10:35:09.642551 mon.0 10.252.24.25:6789/0 50 : [INF] pgmap v28988: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 1848B/s wr, 0op/s -20058> 2013-04-03 10:35:09.642560 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24ebc00 -20057> 2013-04-03 10:35:09.642608 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos -20056> 2013-04-03 10:35:09.642625 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos version 393993 summary v 393993 -20055> 2013-04-03 10:35:09.642644 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -20054> 2013-04-03 10:35:09.642665 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24ebc00 con 0x1bac160 -20053> 2013-04-03 10:35:09.642686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos -20052> 2013-04-03 10:35:09.642701 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos version 393993 summary v 393993 -20051> 2013-04-03 10:35:09.642708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -20050> 2013-04-03 10:35:09.642720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 preprocess_log log(1 entries) v1 from mon.0 -20049> 2013-04-03 10:35:09.642738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -20048> 2013-04-03 10:35:09.642749 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 prepare_log log(1 entries) v1 from mon.0 -20047> 2013-04-03 10:35:09.642759 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 logging 2013-04-03 10:35:09.642551 mon.0 10.252.24.25:6789/0 50 : [INF] pgmap v28988: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 1848B/s wr, 0op/s -20046> 2013-04-03 10:35:09.654346 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113073 ==== paxos(lease_ack lc 427259 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (2438704292 0 0) 0x20f4500 con 0x208a840 -20045> 2013-04-03 10:35:09.654382 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos -20044> 2013-04-03 10:35:09.654398 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393993 update_from_paxos version 393993 summary v 393993 -20043> 2013-04-03 10:35:09.654906 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -20042> 2013-04-03 10:35:09.726932 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113074 ==== forward(pg_stats(1 pgs tid 30436 v 602) v1) to leader v1 ==== 757+0+0 (2362490617 0 0) 0x20e4600 con 0x208a840 -20041> 2013-04-03 10:35:09.726945 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -20040> 2013-04-03 10:35:09.726952 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -20039> 2013-04-03 10:35:09.726981 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 preprocess_query pg_stats(1 pgs tid 30436 v 602) v1 from osd.3 10.252.24.88:6801/17538 -20038> 2013-04-03 10:35:09.726988 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 prepare_update pg_stats(1 pgs tid 30436 v 602) v1 from osd.3 10.252.24.88:6801/17538 -20037> 2013-04-03 10:35:09.726991 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 prepare_pg_stats pg_stats(1 pgs tid 30436 v 602) v1 from osd.3 -20036> 2013-04-03 10:35:09.726996 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28988 got osd.3 osd_stat(1191 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1190 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -20035> 2013-04-03 10:35:10.642182 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393993 encode_pending v393994 -20034> 2013-04-03 10:35:10.642209 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427223..427259) queue_proposal bl 443 bytes; ctx = 0x1b918b0 -20033> 2013-04-03 10:35:10.642216 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427223..427259) propose_queued 427260 443 bytes -20032> 2013-04-03 10:35:10.643381 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427259 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dca00 -20031> 2013-04-03 10:35:10.643446 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28988 encode_full pgmap v 28988 -20030> 2013-04-03 10:35:10.644063 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28988 encode_pending v 28989 -20029> 2013-04-03 10:35:10.644101 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427223..427259) queue_proposal bl 90611 bytes; ctx = 0x1b91a90 -20028> 2013-04-03 10:35:10.644110 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427223..427259) propose_new_value not active; proposal queued -20027> 2013-04-03 10:35:10.722182 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113075 ==== paxos(accept lc 427259 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1983458788 0 0) 0x26dca00 con 0x208a840 -20026> 2013-04-03 10:35:10.722477 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427260 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4500 -20025> 2013-04-03 10:35:10.722512 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427260 fc 427223 pn 0 opn 0) v3 -- ?+0 0x26dc280 -20024> 2013-04-03 10:35:10.722552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos -20023> 2013-04-03 10:35:10.722578 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos version 393994 summary v 393993 -20022> 2013-04-03 10:35:10.722592 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos latest full 393798 -20021> 2013-04-03 10:35:10.722610 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393994 update_from_paxos applying incremental log 393994 2013-04-03 10:35:09.642551 mon.0 10.252.24.25:6789/0 50 : [INF] pgmap v28988: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 1848B/s wr, 0op/s -20020> 2013-04-03 10:35:10.722653 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 check_subs -20019> 2013-04-03 10:35:10.722683 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 create_pending v 393995 -20018> 2013-04-03 10:35:10.722704 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393994 _updated_log for mon.0 10.252.24.25:6789/0 -20017> 2013-04-03 10:35:10.722718 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 50) v1 -- ?+0 0x28a5e00 con 0x1bac160 -20016> 2013-04-03 10:35:10.722742 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427223..427260) propose_queued 427261 90611 bytes -20015> 2013-04-03 10:35:10.725112 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427260 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2799780 -20014> 2013-04-03 10:35:10.725133 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 50) v1 ==== 0+0+0 (0 0 0) 0x28a5e00 con 0x1bac160 -20013> 2013-04-03 10:35:10.725140 7fcdffcdb700 10 handle_log_ack log(last 50) v1 -20012> 2013-04-03 10:35:10.725142 7fcdffcdb700 10 logged 2013-04-03 10:35:09.642551 mon.0 10.252.24.25:6789/0 50 : [INF] pgmap v28988: 240 pgs: 240 active+clean; 252 MB data, 5667 MB used, 1067 GB / 1079 GB avail; 1848B/s wr, 0op/s -20011> 2013-04-03 10:35:10.748356 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113076 ==== paxos(lease_ack lc 427260 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (259732394 0 0) 0x2799780 con 0x208a840 -20010> 2013-04-03 10:35:10.774263 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113077 ==== paxos(accept lc 427260 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (73630864 0 0) 0x26dc280 con 0x208a840 -20009> 2013-04-03 10:35:10.775858 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427261 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2799780 -20008> 2013-04-03 10:35:10.775877 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427261 fc 427223 pn 0 opn 0) v3 -- ?+0 0x26dca00 -20007> 2013-04-03 10:35:10.776000 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28988 update_from_paxos applying incremental 28989 -20006> 2013-04-03 10:35:10.776018 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 v28989: 240 pgs: 240 active+clean; 252 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s -20005> 2013-04-03 10:35:10.776659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 send_pg_creates to 0 pgs -20004> 2013-04-03 10:35:10.776664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 update_logger -20003> 2013-04-03 10:35:10.776673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 create_pending v 28990 -20002> 2013-04-03 10:35:10.776682 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28989 _updated_stats for osd.3 10.252.24.88:6801/17538 -20001> 2013-04-03 10:35:10.776688 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30436) v1 tid 17441) v2 -- ?+0 0x24f8900 con 0x208a840 -20000> 2013-04-03 10:35:10.776698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 check_osd_map already seen 602 >= 602 -19999> 2013-04-03 10:35:10.776702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 update_logger -19998> 2013-04-03 10:35:10.776712 7fcdffcdb700 0 log [INF] : pgmap v28989: 240 pgs: 240 active+clean; 252 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s -19997> 2013-04-03 10:35:10.776713 7fcdffcdb700 10 send_log to self -19996> 2013-04-03 10:35:10.776714 7fcdffcdb700 10 log_queue is 1 last_log 51 sent 50 num 1 unsent 1 sending 1 -19995> 2013-04-03 10:35:10.776715 7fcdffcdb700 10 will send 2013-04-03 10:35:10.776713 mon.0 10.252.24.25:6789/0 51 : [INF] pgmap v28989: 240 pgs: 240 active+clean; 252 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s -19994> 2013-04-03 10:35:10.776723 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2888000 -19993> 2013-04-03 10:35:10.776769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos -19992> 2013-04-03 10:35:10.776785 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos version 393994 summary v 393994 -19991> 2013-04-03 10:35:10.776803 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19990> 2013-04-03 10:35:10.776823 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2888000 con 0x1bac160 -19989> 2013-04-03 10:35:10.776844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos -19988> 2013-04-03 10:35:10.776858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos version 393994 summary v 393994 -19987> 2013-04-03 10:35:10.776867 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19986> 2013-04-03 10:35:10.776877 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 preprocess_log log(1 entries) v1 from mon.0 -19985> 2013-04-03 10:35:10.776894 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19984> 2013-04-03 10:35:10.776906 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 prepare_log log(1 entries) v1 from mon.0 -19983> 2013-04-03 10:35:10.776915 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 logging 2013-04-03 10:35:10.776713 mon.0 10.252.24.25:6789/0 51 : [INF] pgmap v28989: 240 pgs: 240 active+clean; 252 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s -19982> 2013-04-03 10:35:10.833715 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113078 ==== paxos(lease_ack lc 427261 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (2175361433 0 0) 0x26dca00 con 0x208a840 -19981> 2013-04-03 10:35:10.833759 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos -19980> 2013-04-03 10:35:10.833777 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393994 update_from_paxos version 393994 summary v 393994 -19979> 2013-04-03 10:35:10.833792 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19978> 2013-04-03 10:35:11.577760 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 23 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140311 v187) v2 ==== 289+0+0 (2175235643 0 0) 0x28d48c0 con 0x2118420 -19977> 2013-04-03 10:35:11.577831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140311 v187) v2 from mds.? 10.252.58.31:6800/10908 -19976> 2013-04-03 10:35:11.577843 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140311 v187) v2 -- ?+0 0x20e4600 con 0x2118420 -19975> 2013-04-03 10:35:11.774443 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393994 encode_pending v393995 -19974> 2013-04-03 10:35:11.774472 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427223..427261) queue_proposal bl 442 bytes; ctx = 0x1b91aa0 -19973> 2013-04-03 10:35:11.774480 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427223..427261) propose_queued 427262 442 bytes -19972> 2013-04-03 10:35:11.775387 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427261 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dcc80 -19971> 2013-04-03 10:35:11.803580 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113079 ==== paxos(accept lc 427261 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1181817072 0 0) 0x26dcc80 con 0x208a840 -19970> 2013-04-03 10:35:11.803911 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427262 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dca00 -19969> 2013-04-03 10:35:11.803942 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427262 fc 427223 pn 0 opn 0) v3 -- ?+0 0x26dc280 -19968> 2013-04-03 10:35:11.803977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos -19967> 2013-04-03 10:35:11.804003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393994 -19966> 2013-04-03 10:35:11.804017 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos latest full 393798 -19965> 2013-04-03 10:35:11.804034 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393995 update_from_paxos applying incremental log 393995 2013-04-03 10:35:10.776713 mon.0 10.252.24.25:6789/0 51 : [INF] pgmap v28989: 240 pgs: 240 active+clean; 252 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s -19964> 2013-04-03 10:35:11.804077 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 check_subs -19963> 2013-04-03 10:35:11.804106 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 create_pending v 393996 -19962> 2013-04-03 10:35:11.804127 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393995 _updated_log for mon.0 10.252.24.25:6789/0 -19961> 2013-04-03 10:35:11.804140 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 51) v1 -- ?+0 0x28a5e00 con 0x1bac160 -19960> 2013-04-03 10:35:11.804188 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos -19959> 2013-04-03 10:35:11.804203 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 -19958> 2013-04-03 10:35:11.804220 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19957> 2013-04-03 10:35:11.804242 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 51) v1 ==== 0+0+0 (0 0 0) 0x28a5e00 con 0x1bac160 -19956> 2013-04-03 10:35:11.804251 7fcdffcdb700 10 handle_log_ack log(last 51) v1 -19955> 2013-04-03 10:35:11.804252 7fcdffcdb700 10 logged 2013-04-03 10:35:10.776713 mon.0 10.252.24.25:6789/0 51 : [INF] pgmap v28989: 240 pgs: 240 active+clean; 252 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s -19954> 2013-04-03 10:35:11.815964 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113080 ==== paxos(lease_ack lc 427262 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (4234342949 0 0) 0x26dc280 con 0x208a840 -19953> 2013-04-03 10:35:11.815999 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos -19952> 2013-04-03 10:35:11.816015 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 -19951> 2013-04-03 10:35:11.816030 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19950> 2013-04-03 10:35:12.923652 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113081 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 26 v187) v2) to leader v1 ==== 560+0+0 (502391357 0 0) 0x20e3b00 con 0x208a840 -19949> 2013-04-03 10:35:12.923673 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -19948> 2013-04-03 10:35:12.923683 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4b80 from 10.252.24.88:6789/0 -19947> 2013-04-03 10:35:12.923734 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 26 v187) v2 from mds.0 10.252.24.25:6802/3144 -19946> 2013-04-03 10:35:12.923745 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 26 v187) v2 tid 17442) v2 -- ?+0 0x20adb40 con 0x208a840 -19945> 2013-04-03 10:35:13.326117 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28989 check_down_pgs -19944> 2013-04-03 10:35:13.326151 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28989 v28989: 240 pgs: 240 active+clean; 252 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s -19943> 2013-04-03 10:35:13.326177 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -19942> 2013-04-03 10:35:13.326198 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -19941> 2013-04-03 10:35:13.326235 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -19940> 2013-04-03 10:35:13.326264 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos -19939> 2013-04-03 10:35:13.326282 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 -19938> 2013-04-03 10:35:13.326291 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393995 log -19937> 2013-04-03 10:35:13.326307 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19936> 2013-04-03 10:35:13.326325 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -19935> 2013-04-03 10:35:13.377738 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 23 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21682 v187) v2 ==== 289+0+0 (2545713403 0 0) 0x28d32c0 con 0x200d420 -19934> 2013-04-03 10:35:13.377780 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21682 v187) v2 from mds.? 10.252.24.88:6800/17213 -19933> 2013-04-03 10:35:13.377794 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21682 v187) v2 -- ?+0 0x20e3dc0 con 0x200d420 -19932> 2013-04-03 10:35:14.804017 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427262 fc 427223 pn 0 opn 0) v3 -- ?+0 0x26dc000 -19931> 2013-04-03 10:35:14.804794 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113082 ==== paxos(lease_ack lc 427262 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (733754055 0 0) 0x26dc000 con 0x208a840 -19930> 2013-04-03 10:35:14.804858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos -19929> 2013-04-03 10:35:14.804877 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 -19928> 2013-04-03 10:35:14.804894 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19927> 2013-04-03 10:35:15.577878 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 24 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140312 v187) v2 ==== 289+0+0 (2000144209 0 0) 0x28d4600 con 0x2118420 -19926> 2013-04-03 10:35:15.577948 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140312 v187) v2 from mds.? 10.252.58.31:6800/10908 -19925> 2013-04-03 10:35:15.577960 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140312 v187) v2 -- ?+0 0x28d32c0 con 0x2118420 -19924> 2013-04-03 10:35:16.923757 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113083 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 27 v187) v2) to leader v1 ==== 560+0+0 (347618366 0 0) 0x28d48c0 con 0x208a840 -19923> 2013-04-03 10:35:16.923779 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -19922> 2013-04-03 10:35:16.923789 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2146b80 from 10.252.24.88:6789/0 -19921> 2013-04-03 10:35:16.923839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 27 v187) v2 from mds.0 10.252.24.25:6802/3144 -19920> 2013-04-03 10:35:16.923849 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 27 v187) v2 tid 17443) v2 -- ?+0 0x24f8900 con 0x208a840 -19919> 2013-04-03 10:35:17.377881 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 24 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21683 v187) v2 ==== 289+0+0 (3033677702 0 0) 0x28d3000 con 0x200d420 -19918> 2013-04-03 10:35:17.377960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21683 v187) v2 from mds.? 10.252.24.88:6800/17213 -19917> 2013-04-03 10:35:17.377973 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21683 v187) v2 -- ?+0 0x28d48c0 con 0x200d420 -19916> 2013-04-03 10:35:17.804085 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427262 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2909c80 -19915> 2013-04-03 10:35:17.805012 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113084 ==== paxos(lease_ack lc 427262 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (2487196230 0 0) 0x2909c80 con 0x208a840 -19914> 2013-04-03 10:35:17.805059 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos -19913> 2013-04-03 10:35:17.805078 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 -19912> 2013-04-03 10:35:17.805096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19911> 2013-04-03 10:35:18.326472 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28989 v28989: 240 pgs: 240 active+clean; 252 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s -19910> 2013-04-03 10:35:18.326511 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -19909> 2013-04-03 10:35:18.326536 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -19908> 2013-04-03 10:35:18.326581 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -19907> 2013-04-03 10:35:18.326614 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos -19906> 2013-04-03 10:35:18.326632 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 -19905> 2013-04-03 10:35:18.326641 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393995 log -19904> 2013-04-03 10:35:18.326657 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19903> 2013-04-03 10:35:18.326674 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -19902> 2013-04-03 10:35:18.480160 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113085 ==== forward(pg_stats(1 pgs tid 4 v 602) v1) to leader v1 ==== 757+0+0 (1153652132 0 0) 0x28d4600 con 0x208a840 -19901> 2013-04-03 10:35:18.480188 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 -19900> 2013-04-03 10:35:18.480198 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -19899> 2013-04-03 10:35:18.480247 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 preprocess_query pg_stats(1 pgs tid 4 v 602) v1 from osd.2 10.252.24.30:6800/1508 -19898> 2013-04-03 10:35:18.480273 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 prepare_update pg_stats(1 pgs tid 4 v 602) v1 from osd.2 10.252.24.30:6800/1508 -19897> 2013-04-03 10:35:18.480278 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 prepare_pg_stats pg_stats(1 pgs tid 4 v 602) v1 from osd.2 -19896> 2013-04-03 10:35:18.480285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28989 got osd.2 osd_stat(1173 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -19895> 2013-04-03 10:35:18.530468 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28989 encode_pending v 28990 -19894> 2013-04-03 10:35:18.530499 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427223..427262) queue_proposal bl 688 bytes; ctx = 0x1b91a80 -19893> 2013-04-03 10:35:18.530508 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427223..427262) propose_queued 427263 688 bytes -19892> 2013-04-03 10:35:18.531782 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427262 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909a00 -19891> 2013-04-03 10:35:18.558612 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113086 ==== paxos(accept lc 427262 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1959229938 0 0) 0x2909a00 con 0x208a840 -19890> 2013-04-03 10:35:18.558914 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427263 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909c80 -19889> 2013-04-03 10:35:18.558944 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427263 fc 427223 pn 0 opn 0) v3 -- ?+0 0x26dc000 -19888> 2013-04-03 10:35:18.559014 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28989 update_from_paxos applying incremental 28990 -19887> 2013-04-03 10:35:18.559028 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 v28990: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 81380B/s wr, 0op/s -19886> 2013-04-03 10:35:18.559045 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 send_pg_creates to 0 pgs -19885> 2013-04-03 10:35:18.559047 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 update_logger -19884> 2013-04-03 10:35:18.559057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 create_pending v 28991 -19883> 2013-04-03 10:35:18.559065 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28990 _updated_stats for osd.2 10.252.24.30:6800/1508 -19882> 2013-04-03 10:35:18.559070 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 4) v1 tid 17444) v2 -- ?+0 0x20adb40 con 0x208a840 -19881> 2013-04-03 10:35:18.559082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 check_osd_map already seen 602 >= 602 -19880> 2013-04-03 10:35:18.559086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 update_logger -19879> 2013-04-03 10:35:18.559096 7fcdffcdb700 0 log [INF] : pgmap v28990: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 81380B/s wr, 0op/s -19878> 2013-04-03 10:35:18.559097 7fcdffcdb700 10 send_log to self -19877> 2013-04-03 10:35:18.559098 7fcdffcdb700 10 log_queue is 1 last_log 52 sent 51 num 1 unsent 1 sending 1 -19876> 2013-04-03 10:35:18.559100 7fcdffcdb700 10 will send 2013-04-03 10:35:18.559097 mon.0 10.252.24.25:6789/0 52 : [INF] pgmap v28990: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 81380B/s wr, 0op/s -19875> 2013-04-03 10:35:18.559108 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2888000 -19874> 2013-04-03 10:35:18.559156 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos -19873> 2013-04-03 10:35:18.559174 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 -19872> 2013-04-03 10:35:18.559192 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19871> 2013-04-03 10:35:18.559213 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2888000 con 0x1bac160 -19870> 2013-04-03 10:35:18.559234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos -19869> 2013-04-03 10:35:18.559249 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 -19868> 2013-04-03 10:35:18.559257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19867> 2013-04-03 10:35:18.559267 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 preprocess_log log(1 entries) v1 from mon.0 -19866> 2013-04-03 10:35:18.559287 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19865> 2013-04-03 10:35:18.559299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 prepare_log log(1 entries) v1 from mon.0 -19864> 2013-04-03 10:35:18.559308 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 logging 2013-04-03 10:35:18.559097 mon.0 10.252.24.25:6789/0 52 : [INF] pgmap v28990: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 81380B/s wr, 0op/s -19863> 2013-04-03 10:35:18.572283 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113087 ==== paxos(lease_ack lc 427263 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (3195463123 0 0) 0x26dc000 con 0x208a840 -19862> 2013-04-03 10:35:18.572320 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos -19861> 2013-04-03 10:35:18.572463 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393995 update_from_paxos version 393995 summary v 393995 -19860> 2013-04-03 10:35:18.572484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19859> 2013-04-03 10:35:19.559179 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393995 encode_pending v393996 -19858> 2013-04-03 10:35:19.559209 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427223..427263) queue_proposal bl 444 bytes; ctx = 0x1b916f0 -19857> 2013-04-03 10:35:19.559216 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427223..427263) propose_queued 427264 444 bytes -19856> 2013-04-03 10:35:19.560274 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427263 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909780 -19855> 2013-04-03 10:35:19.577985 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 25 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140313 v187) v2 ==== 289+0+0 (1415149612 0 0) 0x28d4340 con 0x2118420 -19854> 2013-04-03 10:35:19.578038 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140313 v187) v2 from mds.? 10.252.58.31:6800/10908 -19853> 2013-04-03 10:35:19.578049 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140313 v187) v2 -- ?+0 0x2145dc0 con 0x2118420 -19852> 2013-04-03 10:35:19.589060 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113088 ==== paxos(accept lc 427263 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3435481232 0 0) 0x2909780 con 0x208a840 -19851> 2013-04-03 10:35:19.589335 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427264 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc000 -19850> 2013-04-03 10:35:19.589366 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427264 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2909a00 -19849> 2013-04-03 10:35:19.589405 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos -19848> 2013-04-03 10:35:19.589431 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393995 -19847> 2013-04-03 10:35:19.589447 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos latest full 393798 -19846> 2013-04-03 10:35:19.589465 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393996 update_from_paxos applying incremental log 393996 2013-04-03 10:35:18.559097 mon.0 10.252.24.25:6789/0 52 : [INF] pgmap v28990: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 81380B/s wr, 0op/s -19845> 2013-04-03 10:35:19.589508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 check_subs -19844> 2013-04-03 10:35:19.589538 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 create_pending v 393997 -19843> 2013-04-03 10:35:19.589559 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393996 _updated_log for mon.0 10.252.24.25:6789/0 -19842> 2013-04-03 10:35:19.589572 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 52) v1 -- ?+0 0x28a5c40 con 0x1bac160 -19841> 2013-04-03 10:35:19.589620 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos -19840> 2013-04-03 10:35:19.589635 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393996 -19839> 2013-04-03 10:35:19.589652 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19838> 2013-04-03 10:35:19.589674 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 52) v1 ==== 0+0+0 (0 0 0) 0x28a5c40 con 0x1bac160 -19837> 2013-04-03 10:35:19.589682 7fcdffcdb700 10 handle_log_ack log(last 52) v1 -19836> 2013-04-03 10:35:19.589683 7fcdffcdb700 10 logged 2013-04-03 10:35:18.559097 mon.0 10.252.24.25:6789/0 52 : [INF] pgmap v28990: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 81380B/s wr, 0op/s -19835> 2013-04-03 10:35:19.601403 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113089 ==== paxos(lease_ack lc 427264 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (1753609618 0 0) 0x2909a00 con 0x208a840 -19834> 2013-04-03 10:35:19.601438 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos -19833> 2013-04-03 10:35:19.601455 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393996 -19832> 2013-04-03 10:35:19.601470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19831> 2013-04-03 10:35:20.923847 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113090 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 28 v187) v2) to leader v1 ==== 560+0+0 (1716612869 0 0) 0x2145840 con 0x208a840 -19830> 2013-04-03 10:35:20.923869 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -19829> 2013-04-03 10:35:20.923878 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2146080 from 10.252.24.88:6789/0 -19828> 2013-04-03 10:35:20.923929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 28 v187) v2 from mds.0 10.252.24.25:6802/3144 -19827> 2013-04-03 10:35:20.923941 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 28 v187) v2 tid 17445) v2 -- ?+0 0x24f8900 con 0x208a840 -19826> 2013-04-03 10:35:21.377995 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 25 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21684 v187) v2 ==== 289+0+0 (1557857269 0 0) 0x20e48c0 con 0x200d420 -19825> 2013-04-03 10:35:21.378066 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21684 v187) v2 from mds.? 10.252.24.88:6800/17213 -19824> 2013-04-03 10:35:21.378078 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21684 v187) v2 -- ?+0 0x2145840 con 0x200d420 -19823> 2013-04-03 10:35:22.589440 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427264 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2909500 -19822> 2013-04-03 10:35:22.590245 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113091 ==== paxos(lease_ack lc 427264 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (2547081178 0 0) 0x2909500 con 0x208a840 -19821> 2013-04-03 10:35:22.590312 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos -19820> 2013-04-03 10:35:22.590330 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393996 -19819> 2013-04-03 10:35:22.590347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19818> 2013-04-03 10:35:23.326815 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28990 check_down_pgs -19817> 2013-04-03 10:35:23.326847 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28990 v28990: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 81380B/s wr, 0op/s -19816> 2013-04-03 10:35:23.326873 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -19815> 2013-04-03 10:35:23.326897 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -19814> 2013-04-03 10:35:23.326933 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -19813> 2013-04-03 10:35:23.326971 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos -19812> 2013-04-03 10:35:23.326989 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393996 -19811> 2013-04-03 10:35:23.326998 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393996 log -19810> 2013-04-03 10:35:23.327014 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19809> 2013-04-03 10:35:23.327033 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -19808> 2013-04-03 10:35:23.578123 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 26 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140314 v187) v2 ==== 289+0+0 (837466539 0 0) 0x28d4080 con 0x2118420 -19807> 2013-04-03 10:35:23.578196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140314 v187) v2 from mds.? 10.252.58.31:6800/10908 -19806> 2013-04-03 10:35:23.578209 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140314 v187) v2 -- ?+0 0x20e48c0 con 0x2118420 -19805> 2013-04-03 10:35:24.560789 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113092 ==== forward(pg_stats(1 pgs tid 12 v 602) v1) to leader v1 ==== 757+0+0 (1581036710 0 0) 0x28d4340 con 0x208a840 -19804> 2013-04-03 10:35:24.560810 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -19803> 2013-04-03 10:35:24.560823 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -19802> 2013-04-03 10:35:24.560874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 preprocess_query pg_stats(1 pgs tid 12 v 602) v1 from osd.1 10.252.24.25:6805/3572 -19801> 2013-04-03 10:35:24.560897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 prepare_update pg_stats(1 pgs tid 12 v 602) v1 from osd.1 10.252.24.25:6805/3572 -19800> 2013-04-03 10:35:24.560903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 prepare_pg_stats pg_stats(1 pgs tid 12 v 602) v1 from osd.1 -19799> 2013-04-03 10:35:24.560909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28990 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -19798> 2013-04-03 10:35:24.611093 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28990 encode_pending v 28991 -19797> 2013-04-03 10:35:24.611125 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427223..427264) queue_proposal bl 654 bytes; ctx = 0x1b916e0 -19796> 2013-04-03 10:35:24.611132 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427223..427264) propose_queued 427265 654 bytes -19795> 2013-04-03 10:35:24.612627 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427264 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909280 -19794> 2013-04-03 10:35:24.669626 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113093 ==== paxos(accept lc 427264 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (329774612 0 0) 0x2909280 con 0x208a840 -19793> 2013-04-03 10:35:24.669934 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427265 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909500 -19792> 2013-04-03 10:35:24.669964 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427265 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2909a00 -19791> 2013-04-03 10:35:24.670036 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28990 update_from_paxos applying incremental 28991 -19790> 2013-04-03 10:35:24.670051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 v28991: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 52200B/s wr, 0op/s -19789> 2013-04-03 10:35:24.670071 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 send_pg_creates to 0 pgs -19788> 2013-04-03 10:35:24.670075 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 update_logger -19787> 2013-04-03 10:35:24.670085 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 create_pending v 28992 -19786> 2013-04-03 10:35:24.670093 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28991 _updated_stats for osd.1 10.252.24.25:6805/3572 -19785> 2013-04-03 10:35:24.670100 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 12) v1 tid 17446) v2 -- ?+0 0x20adb40 con 0x208a840 -19784> 2013-04-03 10:35:24.670109 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 check_osd_map already seen 602 >= 602 -19783> 2013-04-03 10:35:24.670112 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 update_logger -19782> 2013-04-03 10:35:24.670122 7fcdffcdb700 0 log [INF] : pgmap v28991: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 52200B/s wr, 0op/s -19781> 2013-04-03 10:35:24.670124 7fcdffcdb700 10 send_log to self -19780> 2013-04-03 10:35:24.670127 7fcdffcdb700 10 log_queue is 1 last_log 53 sent 52 num 1 unsent 1 sending 1 -19779> 2013-04-03 10:35:24.670129 7fcdffcdb700 10 will send 2013-04-03 10:35:24.670123 mon.0 10.252.24.25:6789/0 53 : [INF] pgmap v28991: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 52200B/s wr, 0op/s -19778> 2013-04-03 10:35:24.670137 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2888000 -19777> 2013-04-03 10:35:24.670186 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos -19776> 2013-04-03 10:35:24.670204 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393996 -19775> 2013-04-03 10:35:24.670222 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19774> 2013-04-03 10:35:24.670244 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2888000 con 0x1bac160 -19773> 2013-04-03 10:35:24.670265 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos -19772> 2013-04-03 10:35:24.670279 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393996 -19771> 2013-04-03 10:35:24.670288 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19770> 2013-04-03 10:35:24.670299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 preprocess_log log(1 entries) v1 from mon.0 -19769> 2013-04-03 10:35:24.670321 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19768> 2013-04-03 10:35:24.670334 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 prepare_log log(1 entries) v1 from mon.0 -19767> 2013-04-03 10:35:24.670345 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 logging 2013-04-03 10:35:24.670123 mon.0 10.252.24.25:6789/0 53 : [INF] pgmap v28991: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 52200B/s wr, 0op/s -19766> 2013-04-03 10:35:24.685006 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113094 ==== paxos(lease_ack lc 427265 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (113391151 0 0) 0x2909a00 con 0x208a840 -19765> 2013-04-03 10:35:24.685041 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos -19764> 2013-04-03 10:35:24.685056 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393996 update_from_paxos version 393996 summary v 393996 -19763> 2013-04-03 10:35:24.685071 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19762> 2013-04-03 10:35:24.923992 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113095 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 29 v187) v2) to leader v1 ==== 560+0+0 (3139725904 0 0) 0x2145b00 con 0x208a840 -19761> 2013-04-03 10:35:24.924008 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -19760> 2013-04-03 10:35:24.924016 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae340 from 10.252.24.88:6789/0 -19759> 2013-04-03 10:35:24.924047 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 29 v187) v2 from mds.0 10.252.24.25:6802/3144 -19758> 2013-04-03 10:35:24.924057 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 29 v187) v2 tid 17447) v2 -- ?+0 0x24f8900 con 0x208a840 -19757> 2013-04-03 10:35:25.378099 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 26 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21685 v187) v2 ==== 289+0+0 (2142616712 0 0) 0x20e3dc0 con 0x200d420 -19756> 2013-04-03 10:35:25.378174 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21685 v187) v2 from mds.? 10.252.24.88:6800/17213 -19755> 2013-04-03 10:35:25.378187 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21685 v187) v2 -- ?+0 0x2145b00 con 0x200d420 -19754> 2013-04-03 10:35:25.669815 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393996 encode_pending v393997 -19753> 2013-04-03 10:35:25.669844 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427223..427265) queue_proposal bl 444 bytes; ctx = 0x1b91c00 -19752> 2013-04-03 10:35:25.669851 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427223..427265) propose_queued 427266 444 bytes -19751> 2013-04-03 10:35:25.671048 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427265 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909000 -19750> 2013-04-03 10:35:25.701758 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113096 ==== paxos(accept lc 427265 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3916732577 0 0) 0x2909000 con 0x208a840 -19749> 2013-04-03 10:35:25.702043 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427266 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909a00 -19748> 2013-04-03 10:35:25.702073 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427266 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2909280 -19747> 2013-04-03 10:35:25.702111 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos -19746> 2013-04-03 10:35:25.702138 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393996 -19745> 2013-04-03 10:35:25.702152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos latest full 393798 -19744> 2013-04-03 10:35:25.702170 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393997 update_from_paxos applying incremental log 393997 2013-04-03 10:35:24.670123 mon.0 10.252.24.25:6789/0 53 : [INF] pgmap v28991: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 52200B/s wr, 0op/s -19743> 2013-04-03 10:35:25.702214 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 check_subs -19742> 2013-04-03 10:35:25.702244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 create_pending v 393998 -19741> 2013-04-03 10:35:25.702266 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393997 _updated_log for mon.0 10.252.24.25:6789/0 -19740> 2013-04-03 10:35:25.702278 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 53) v1 -- ?+0 0x28a5a80 con 0x1bac160 -19739> 2013-04-03 10:35:25.702328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos -19738> 2013-04-03 10:35:25.702347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 -19737> 2013-04-03 10:35:25.702364 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19736> 2013-04-03 10:35:25.702385 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 53) v1 ==== 0+0+0 (0 0 0) 0x28a5a80 con 0x1bac160 -19735> 2013-04-03 10:35:25.702393 7fcdffcdb700 10 handle_log_ack log(last 53) v1 -19734> 2013-04-03 10:35:25.702395 7fcdffcdb700 10 logged 2013-04-03 10:35:24.670123 mon.0 10.252.24.25:6789/0 53 : [INF] pgmap v28991: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 52200B/s wr, 0op/s -19733> 2013-04-03 10:35:25.714123 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113097 ==== paxos(lease_ack lc 427266 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (439103584 0 0) 0x2909280 con 0x208a840 -19732> 2013-04-03 10:35:25.714158 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos -19731> 2013-04-03 10:35:25.714174 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 -19730> 2013-04-03 10:35:25.714188 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19729> 2013-04-03 10:35:27.578247 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 27 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140315 v187) v2 ==== 289+0+0 (310642390 0 0) 0x20e4600 con 0x2118420 -19728> 2013-04-03 10:35:27.578318 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140315 v187) v2 from mds.? 10.252.58.31:6800/10908 -19727> 2013-04-03 10:35:27.578330 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140315 v187) v2 -- ?+0 0x20e3dc0 con 0x2118420 -19726> 2013-04-03 10:35:28.327173 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28991 check_down_pgs -19725> 2013-04-03 10:35:28.327204 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28991 v28991: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 52200B/s wr, 0op/s -19724> 2013-04-03 10:35:28.327230 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -19723> 2013-04-03 10:35:28.327254 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -19722> 2013-04-03 10:35:28.327291 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -19721> 2013-04-03 10:35:28.327325 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos -19720> 2013-04-03 10:35:28.327346 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 -19719> 2013-04-03 10:35:28.327356 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393997 log -19718> 2013-04-03 10:35:28.327372 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19717> 2013-04-03 10:35:28.327390 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -19716> 2013-04-03 10:35:28.702152 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427266 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2959c80 -19715> 2013-04-03 10:35:28.702940 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113098 ==== paxos(lease_ack lc 427266 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (1899390458 0 0) 0x2959c80 con 0x208a840 -19714> 2013-04-03 10:35:28.703005 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos -19713> 2013-04-03 10:35:28.703023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 -19712> 2013-04-03 10:35:28.703040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19711> 2013-04-03 10:35:28.924084 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113099 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 30 v187) v2) to leader v1 ==== 560+0+0 (4265213470 0 0) 0x28d4340 con 0x208a840 -19710> 2013-04-03 10:35:28.924098 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -19709> 2013-04-03 10:35:28.924108 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae600 from 10.252.24.88:6789/0 -19708> 2013-04-03 10:35:28.924142 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 30 v187) v2 from mds.0 10.252.24.25:6802/3144 -19707> 2013-04-03 10:35:28.924153 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 30 v187) v2 tid 17448) v2 -- ?+0 0x20adb40 con 0x208a840 -19706> 2013-04-03 10:35:29.378188 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 27 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21686 v187) v2 ==== 289+0+0 (436632847 0 0) 0x20e4080 con 0x200d420 -19705> 2013-04-03 10:35:29.378257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21686 v187) v2 from mds.? 10.252.24.88:6800/17213 -19704> 2013-04-03 10:35:29.378272 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21686 v187) v2 -- ?+0 0x28d4340 con 0x200d420 -19703> 2013-04-03 10:35:31.578405 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 28 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140316 v187) v2 ==== 289+0+0 (4203559589 0 0) 0x20e3840 con 0x2118420 -19702> 2013-04-03 10:35:31.578479 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140316 v187) v2 from mds.? 10.252.58.31:6800/10908 -19701> 2013-04-03 10:35:31.578491 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140316 v187) v2 -- ?+0 0x20e4080 con 0x2118420 -19700> 2013-04-03 10:35:31.702214 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427266 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2959a00 -19699> 2013-04-03 10:35:31.702989 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113100 ==== paxos(lease_ack lc 427266 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (1911702604 0 0) 0x2959a00 con 0x208a840 -19698> 2013-04-03 10:35:31.703033 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos -19697> 2013-04-03 10:35:31.703051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 -19696> 2013-04-03 10:35:31.703068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19695> 2013-04-03 10:35:32.924282 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113101 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 31 v187) v2) to leader v1 ==== 560+0+0 (4151320605 0 0) 0x20e4600 con 0x208a840 -19694> 2013-04-03 10:35:32.924302 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -19693> 2013-04-03 10:35:32.924311 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x295eb80 from 10.252.24.88:6789/0 -19692> 2013-04-03 10:35:32.924362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 31 v187) v2 from mds.0 10.252.24.25:6802/3144 -19691> 2013-04-03 10:35:32.924374 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 31 v187) v2 tid 17449) v2 -- ?+0 0x24f8900 con 0x208a840 -19690> 2013-04-03 10:35:33.327529 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28991 v28991: 240 pgs: 240 active+clean; 253 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 52200B/s wr, 0op/s -19689> 2013-04-03 10:35:33.327570 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -19688> 2013-04-03 10:35:33.327592 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -19687> 2013-04-03 10:35:33.327638 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -19686> 2013-04-03 10:35:33.327669 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos -19685> 2013-04-03 10:35:33.327686 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 -19684> 2013-04-03 10:35:33.327695 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393997 log -19683> 2013-04-03 10:35:33.327712 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19682> 2013-04-03 10:35:33.327730 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -19681> 2013-04-03 10:35:33.378285 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 28 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21687 v187) v2 ==== 289+0+0 (963166834 0 0) 0x2145580 con 0x200d420 -19680> 2013-04-03 10:35:33.378329 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21687 v187) v2 from mds.? 10.252.24.88:6800/17213 -19679> 2013-04-03 10:35:33.378346 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21687 v187) v2 -- ?+0 0x20e4600 con 0x200d420 -19678> 2013-04-03 10:35:33.482223 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113102 ==== forward(pg_stats(1 pgs tid 5 v 602) v1) to leader v1 ==== 757+0+0 (3493339144 0 0) 0x20e3840 con 0x208a840 -19677> 2013-04-03 10:35:33.482237 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 -19676> 2013-04-03 10:35:33.482245 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -19675> 2013-04-03 10:35:33.482275 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 preprocess_query pg_stats(1 pgs tid 5 v 602) v1 from osd.2 10.252.24.30:6800/1508 -19674> 2013-04-03 10:35:33.482296 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 prepare_update pg_stats(1 pgs tid 5 v 602) v1 from osd.2 10.252.24.30:6800/1508 -19673> 2013-04-03 10:35:33.482301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 prepare_pg_stats pg_stats(1 pgs tid 5 v 602) v1 from osd.2 -19672> 2013-04-03 10:35:33.482307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28991 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1173 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -19671> 2013-04-03 10:35:33.532491 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28991 encode_pending v 28992 -19670> 2013-04-03 10:35:33.532535 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427223..427266) queue_proposal bl 654 bytes; ctx = 0x1b91a60 -19669> 2013-04-03 10:35:33.532546 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427223..427266) propose_queued 427267 654 bytes -19668> 2013-04-03 10:35:33.533465 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427266 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2959780 -19667> 2013-04-03 10:35:33.579715 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113103 ==== paxos(accept lc 427266 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4147247767 0 0) 0x2959780 con 0x208a840 -19666> 2013-04-03 10:35:33.580088 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427267 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2959a00 -19665> 2013-04-03 10:35:33.580110 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427267 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2959c80 -19664> 2013-04-03 10:35:33.580201 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28991 update_from_paxos applying incremental 28992 -19663> 2013-04-03 10:35:33.580221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28992 v28992: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19662> 2013-04-03 10:35:33.580239 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28992 send_pg_creates to 0 pgs -19661> 2013-04-03 10:35:33.580243 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28992 update_logger -19660> 2013-04-03 10:35:33.580254 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28992 create_pending v 28993 -19659> 2013-04-03 10:35:33.580263 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28992 _updated_stats for osd.2 10.252.24.30:6800/1508 -19658> 2013-04-03 10:35:33.580268 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 5) v1 tid 17450) v2 -- ?+0 0x20adb40 con 0x208a840 -19657> 2013-04-03 10:35:33.580279 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28992 check_osd_map already seen 602 >= 602 -19656> 2013-04-03 10:35:33.580285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28992 update_logger -19655> 2013-04-03 10:35:33.580295 7fcdffcdb700 0 log [INF] : pgmap v28992: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19654> 2013-04-03 10:35:33.580296 7fcdffcdb700 10 send_log to self -19653> 2013-04-03 10:35:33.580297 7fcdffcdb700 10 log_queue is 1 last_log 54 sent 53 num 1 unsent 1 sending 1 -19652> 2013-04-03 10:35:33.580299 7fcdffcdb700 10 will send 2013-04-03 10:35:33.580296 mon.0 10.252.24.25:6789/0 54 : [INF] pgmap v28992: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19651> 2013-04-03 10:35:33.580306 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2888000 -19650> 2013-04-03 10:35:33.580356 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos -19649> 2013-04-03 10:35:33.580374 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 -19648> 2013-04-03 10:35:33.580392 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19647> 2013-04-03 10:35:33.580413 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2888000 con 0x1bac160 -19646> 2013-04-03 10:35:33.580434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos -19645> 2013-04-03 10:35:33.580449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 -19644> 2013-04-03 10:35:33.580458 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19643> 2013-04-03 10:35:33.580469 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 preprocess_log log(1 entries) v1 from mon.0 -19642> 2013-04-03 10:35:33.580489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19641> 2013-04-03 10:35:33.580501 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 prepare_log log(1 entries) v1 from mon.0 -19640> 2013-04-03 10:35:33.580510 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 logging 2013-04-03 10:35:33.580296 mon.0 10.252.24.25:6789/0 54 : [INF] pgmap v28992: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19639> 2013-04-03 10:35:33.605735 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113104 ==== paxos(lease_ack lc 427267 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (591591920 0 0) 0x2959c80 con 0x208a840 -19638> 2013-04-03 10:35:33.605776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos -19637> 2013-04-03 10:35:33.605794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393997 update_from_paxos version 393997 summary v 393997 -19636> 2013-04-03 10:35:33.605808 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19635> 2013-04-03 10:35:34.579923 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393997 encode_pending v393998 -19634> 2013-04-03 10:35:34.579954 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427223..427267) queue_proposal bl 443 bytes; ctx = 0x1b91c50 -19633> 2013-04-03 10:35:34.579961 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427223..427267) propose_queued 427268 443 bytes -19632> 2013-04-03 10:35:34.581091 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427267 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2959500 -19631> 2013-04-03 10:35:34.608997 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113105 ==== paxos(accept lc 427267 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4055893893 0 0) 0x2959500 con 0x208a840 -19630> 2013-04-03 10:35:34.609279 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427268 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2959c80 -19629> 2013-04-03 10:35:34.609312 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427268 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2959780 -19628> 2013-04-03 10:35:34.609353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos -19627> 2013-04-03 10:35:34.609379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos version 393998 summary v 393997 -19626> 2013-04-03 10:35:34.609394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos latest full 393798 -19625> 2013-04-03 10:35:34.609412 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393998 update_from_paxos applying incremental log 393998 2013-04-03 10:35:33.580296 mon.0 10.252.24.25:6789/0 54 : [INF] pgmap v28992: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19624> 2013-04-03 10:35:34.609462 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 check_subs -19623> 2013-04-03 10:35:34.609493 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 create_pending v 393999 -19622> 2013-04-03 10:35:34.609514 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v393998 _updated_log for mon.0 10.252.24.25:6789/0 -19621> 2013-04-03 10:35:34.609527 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 54) v1 -- ?+0 0x28a58c0 con 0x1bac160 -19620> 2013-04-03 10:35:34.609575 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos -19619> 2013-04-03 10:35:34.609590 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos version 393998 summary v 393998 -19618> 2013-04-03 10:35:34.609608 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19617> 2013-04-03 10:35:34.609629 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 54) v1 ==== 0+0+0 (0 0 0) 0x28a58c0 con 0x1bac160 -19616> 2013-04-03 10:35:34.609637 7fcdffcdb700 10 handle_log_ack log(last 54) v1 -19615> 2013-04-03 10:35:34.609639 7fcdffcdb700 10 logged 2013-04-03 10:35:33.580296 mon.0 10.252.24.25:6789/0 54 : [INF] pgmap v28992: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19614> 2013-04-03 10:35:34.621374 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113106 ==== paxos(lease_ack lc 427268 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (2295585150 0 0) 0x2959780 con 0x208a840 -19613> 2013-04-03 10:35:34.621412 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos -19612> 2013-04-03 10:35:34.621428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos version 393998 summary v 393998 -19611> 2013-04-03 10:35:34.621442 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19610> 2013-04-03 10:35:35.578515 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 29 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140317 v187) v2 ==== 289+0+0 (3655566808 0 0) 0x20e3b00 con 0x2118420 -19609> 2013-04-03 10:35:35.578588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140317 v187) v2 from mds.? 10.252.58.31:6800/10908 -19608> 2013-04-03 10:35:35.578601 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140317 v187) v2 -- ?+0 0x20e3840 con 0x2118420 -19607> 2013-04-03 10:35:36.924403 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113107 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 32 v187) v2) to leader v1 ==== 560+0+0 (1470729225 0 0) 0x295e8c0 con 0x208a840 -19606> 2013-04-03 10:35:36.924423 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -19605> 2013-04-03 10:35:36.924432 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x295e600 from 10.252.24.88:6789/0 -19604> 2013-04-03 10:35:36.924483 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 32 v187) v2 from mds.0 10.252.24.25:6802/3144 -19603> 2013-04-03 10:35:36.924494 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 32 v187) v2 tid 17451) v2 -- ?+0 0x24f8900 con 0x208a840 -19602> 2013-04-03 10:35:37.378417 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 29 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21688 v187) v2 ==== 289+0+0 (3489166104 0 0) 0x2146b80 con 0x200d420 -19601> 2013-04-03 10:35:37.378492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21688 v187) v2 from mds.? 10.252.24.88:6800/17213 -19600> 2013-04-03 10:35:37.378506 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21688 v187) v2 -- ?+0 0x295e8c0 con 0x200d420 -19599> 2013-04-03 10:35:37.609394 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427268 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2959280 -19598> 2013-04-03 10:35:37.610311 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113108 ==== paxos(lease_ack lc 427268 fc 427223 pn 0 opn 0) v3 ==== 80+0+0 (472157533 0 0) 0x2959280 con 0x208a840 -19597> 2013-04-03 10:35:37.610375 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos -19596> 2013-04-03 10:35:37.610394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos version 393998 summary v 393998 -19595> 2013-04-03 10:35:37.610411 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19594> 2013-04-03 10:35:38.157601 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427223..427268) queue_proposal bl 1107 bytes; ctx = 0x1b91c20 -19593> 2013-04-03 10:35:38.327879 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28992 check_down_pgs -19592> 2013-04-03 10:35:38.327900 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28992 v28992: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19591> 2013-04-03 10:35:38.327925 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -19590> 2013-04-03 10:35:38.327949 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -19589> 2013-04-03 10:35:38.327985 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -19588> 2013-04-03 10:35:38.328015 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos -19587> 2013-04-03 10:35:38.328033 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393998 update_from_paxos version 393998 summary v 393998 -19586> 2013-04-03 10:35:38.328042 7fce004dc700 10 mon.hotcpc9039@0(leader).log v393998 log -19585> 2013-04-03 10:35:38.328059 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19584> 2013-04-03 10:35:38.328076 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -19583> 2013-04-03 10:35:38.627372 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113109 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (687457687 0 0) 0x20e3b00 con 0x208a840 -19582> 2013-04-03 10:35:38.627402 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -19581> 2013-04-03 10:35:38.627409 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -19580> 2013-04-03 10:35:38.627420 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 -19579> 2013-04-03 10:35:38.627424 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( start_reply flags( reply-to(mon.2 10.252.24.88:6789/0) ) ) v1 -- ?+0 0x2146b80 -19578> 2013-04-03 10:35:39.562562 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113110 ==== forward(pg_stats(1 pgs tid 13 v 602) v1) to leader v1 ==== 757+0+0 (2433153278 0 0) 0x295e340 con 0x208a840 -19577> 2013-04-03 10:35:39.562584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -19576> 2013-04-03 10:35:39.562594 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -19575> 2013-04-03 10:35:39.562647 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28992 preprocess_query pg_stats(1 pgs tid 13 v 602) v1 from osd.1 10.252.24.25:6805/3572 -19574> 2013-04-03 10:35:39.562671 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28992 prepare_update pg_stats(1 pgs tid 13 v 602) v1 from osd.1 10.252.24.25:6805/3572 -19573> 2013-04-03 10:35:39.562678 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28992 prepare_pg_stats pg_stats(1 pgs tid 13 v 602) v1 from osd.1 -19572> 2013-04-03 10:35:39.562684 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28992 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -19571> 2013-04-03 10:35:39.578617 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 30 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140318 v187) v2 ==== 289+0+0 (3159376991 0 0) 0x20e4b80 con 0x2118420 -19570> 2013-04-03 10:35:39.578866 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140318 v187) v2 from mds.? 10.252.58.31:6800/10908 -19569> 2013-04-03 10:35:39.578885 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140318 v187) v2 -- ?+0 0x295e340 con 0x2118420 -19568> 2013-04-03 10:35:39.612893 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28992 encode_pending v 28993 -19567> 2013-04-03 10:35:39.612940 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427223..427268) queue_proposal bl 654 bytes; ctx = 0x1b91c30 -19566> 2013-04-03 10:35:39.612949 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427223..427268) propose_queued 427269 1107 bytes -19565> 2013-04-03 10:35:39.615584 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427268 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2959000 -19564> 2013-04-03 10:35:39.647166 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113111 ==== paxos(accept lc 427268 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3784118857 0 0) 0x2959000 con 0x208a840 -19563> 2013-04-03 10:35:39.658104 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427269 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2959280 -19562> 2013-04-03 10:35:39.658128 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427269 fc 427223 pn 0 opn 0) v3 -- ?+0 0x2959780 -19561> 2013-04-03 10:35:39.658190 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427258..427269) propose_queued 427270 654 bytes -19560> 2013-04-03 10:35:39.658420 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427269 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2959500 -19559> 2013-04-03 10:35:39.681399 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113112 ==== paxos(lease_ack lc 427269 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (3141495485 0 0) 0x2959500 con 0x208a840 -19558> 2013-04-03 10:35:39.693579 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113113 ==== paxos(accept lc 427269 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3013524088 0 0) 0x2959780 con 0x208a840 -19557> 2013-04-03 10:35:39.693852 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427270 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2959500 -19556> 2013-04-03 10:35:39.693887 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427270 fc 427258 pn 0 opn 0) v3 -- ?+0 0x2959000 -19555> 2013-04-03 10:35:39.693952 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28992 update_from_paxos applying incremental 28993 -19554> 2013-04-03 10:35:39.693971 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 v28993: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19553> 2013-04-03 10:35:39.693989 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 send_pg_creates to 0 pgs -19552> 2013-04-03 10:35:39.693992 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 update_logger -19551> 2013-04-03 10:35:39.694002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 create_pending v 28994 -19550> 2013-04-03 10:35:39.694011 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 _updated_stats for osd.1 10.252.24.25:6805/3572 -19549> 2013-04-03 10:35:39.694019 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 13) v1 tid 17452) v2 -- ?+0 0x20adb40 con 0x208a840 -19548> 2013-04-03 10:35:39.694028 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 check_osd_map already seen 602 >= 602 -19547> 2013-04-03 10:35:39.694032 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 update_logger -19546> 2013-04-03 10:35:39.694042 7fcdffcdb700 0 log [INF] : pgmap v28993: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19545> 2013-04-03 10:35:39.694044 7fcdffcdb700 10 send_log to self -19544> 2013-04-03 10:35:39.694044 7fcdffcdb700 10 log_queue is 1 last_log 55 sent 54 num 1 unsent 1 sending 1 -19543> 2013-04-03 10:35:39.694046 7fcdffcdb700 10 will send 2013-04-03 10:35:39.694043 mon.0 10.252.24.25:6789/0 55 : [INF] pgmap v28993: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19542> 2013-04-03 10:35:39.694054 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2888000 -19541> 2013-04-03 10:35:39.694109 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 update_from_paxos -19540> 2013-04-03 10:35:39.694128 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 update_from_paxos version 393998 summary v 393998 -19539> 2013-04-03 10:35:39.694147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19538> 2013-04-03 10:35:39.694169 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2888000 con 0x1bac160 -19537> 2013-04-03 10:35:39.694190 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 update_from_paxos -19536> 2013-04-03 10:35:39.694205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 update_from_paxos version 393998 summary v 393998 -19535> 2013-04-03 10:35:39.694215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19534> 2013-04-03 10:35:39.694227 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 preprocess_log log(1 entries) v1 from mon.0 -19533> 2013-04-03 10:35:39.694247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19532> 2013-04-03 10:35:39.694259 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 prepare_log log(1 entries) v1 from mon.0 -19531> 2013-04-03 10:35:39.694270 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 logging 2013-04-03 10:35:39.694043 mon.0 10.252.24.25:6789/0 55 : [INF] pgmap v28993: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19530> 2013-04-03 10:35:39.707366 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113114 ==== paxos(lease_ack lc 427270 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (1429203613 0 0) 0x2959000 con 0x208a840 -19529> 2013-04-03 10:35:39.707400 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 update_from_paxos -19528> 2013-04-03 10:35:39.707415 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 update_from_paxos version 393998 summary v 393998 -19527> 2013-04-03 10:35:39.707431 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19526> 2013-04-03 10:35:39.729970 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113115 ==== forward(pg_stats(1 pgs tid 30437 v 602) v1) to leader v1 ==== 757+0+0 (224731796 0 0) 0x295e080 con 0x208a840 -19525> 2013-04-03 10:35:39.729982 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -19524> 2013-04-03 10:35:39.729990 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -19523> 2013-04-03 10:35:39.730017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 preprocess_query pg_stats(1 pgs tid 30437 v 602) v1 from osd.3 10.252.24.88:6801/17538 -19522> 2013-04-03 10:35:39.730039 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 prepare_update pg_stats(1 pgs tid 30437 v 602) v1 from osd.3 10.252.24.88:6801/17538 -19521> 2013-04-03 10:35:39.730046 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 prepare_pg_stats pg_stats(1 pgs tid 30437 v 602) v1 from osd.3 -19520> 2013-04-03 10:35:39.730051 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1191 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -19519> 2013-04-03 10:35:40.693773 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393998 encode_pending v393999 -19518> 2013-04-03 10:35:40.693803 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427258..427270) queue_proposal bl 443 bytes; ctx = 0x1b91ab0 -19517> 2013-04-03 10:35:40.693810 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427258..427270) propose_queued 427271 443 bytes -19516> 2013-04-03 10:35:40.694895 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427270 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2981c80 -19515> 2013-04-03 10:35:40.694988 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 encode_pending v 28994 -19514> 2013-04-03 10:35:40.695022 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427258..427270) queue_proposal bl 654 bytes; ctx = 0x1b91a40 -19513> 2013-04-03 10:35:40.695030 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427258..427270) propose_new_value not active; proposal queued -19512> 2013-04-03 10:35:40.849053 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113116 ==== paxos(accept lc 427270 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1091021099 0 0) 0x2981c80 con 0x208a840 -19511> 2013-04-03 10:35:40.849411 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427271 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2959000 -19510> 2013-04-03 10:35:40.849441 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427271 fc 427258 pn 0 opn 0) v3 -- ?+0 0x2959780 -19509> 2013-04-03 10:35:40.849488 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos -19508> 2013-04-03 10:35:40.849526 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos version 393999 summary v 393998 -19507> 2013-04-03 10:35:40.849541 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos latest full 393798 -19506> 2013-04-03 10:35:40.849559 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos applying incremental log 393999 2013-04-03 10:35:39.694043 mon.0 10.252.24.25:6789/0 55 : [INF] pgmap v28993: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19505> 2013-04-03 10:35:40.849608 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 check_subs -19504> 2013-04-03 10:35:40.849639 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 create_pending v 394000 -19503> 2013-04-03 10:35:40.849661 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 _updated_log for mon.0 10.252.24.25:6789/0 -19502> 2013-04-03 10:35:40.849674 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 55) v1 -- ?+0 0x28a5540 con 0x1bac160 -19501> 2013-04-03 10:35:40.849698 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427258..427271) propose_queued 427272 654 bytes -19500> 2013-04-03 10:35:40.857994 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427271 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909280 -19499> 2013-04-03 10:35:40.858028 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 55) v1 ==== 0+0+0 (0 0 0) 0x28a5540 con 0x1bac160 -19498> 2013-04-03 10:35:40.858045 7fcdffcdb700 10 handle_log_ack log(last 55) v1 -19497> 2013-04-03 10:35:40.858047 7fcdffcdb700 10 logged 2013-04-03 10:35:39.694043 mon.0 10.252.24.25:6789/0 55 : [INF] pgmap v28993: 240 pgs: 240 active+clean; 253 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19496> 2013-04-03 10:35:40.861322 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113117 ==== paxos(lease_ack lc 427271 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (2895827708 0 0) 0x2909280 con 0x208a840 -19495> 2013-04-03 10:35:40.874925 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113118 ==== paxos(accept lc 427271 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2910652094 0 0) 0x2909a00 con 0x208a840 -19494> 2013-04-03 10:35:40.876356 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427272 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909280 -19493> 2013-04-03 10:35:40.876393 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427272 fc 427258 pn 0 opn 0) v3 -- ?+0 0x2981c80 -19492> 2013-04-03 10:35:40.876461 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28993 update_from_paxos applying incremental 28994 -19491> 2013-04-03 10:35:40.876481 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 v28994: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 714B/s wr, 0op/s -19490> 2013-04-03 10:35:40.876501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 send_pg_creates to 0 pgs -19489> 2013-04-03 10:35:40.876506 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 update_logger -19488> 2013-04-03 10:35:40.876516 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 create_pending v 28995 -19487> 2013-04-03 10:35:40.876525 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 _updated_stats for osd.3 10.252.24.88:6801/17538 -19486> 2013-04-03 10:35:40.876533 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30437) v1 tid 17453) v2 -- ?+0 0x24f8900 con 0x208a840 -19485> 2013-04-03 10:35:40.876542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 check_osd_map already seen 602 >= 602 -19484> 2013-04-03 10:35:40.876546 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 update_logger -19483> 2013-04-03 10:35:40.876557 7fcdffcdb700 0 log [INF] : pgmap v28994: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 714B/s wr, 0op/s -19482> 2013-04-03 10:35:40.876558 7fcdffcdb700 10 send_log to self -19481> 2013-04-03 10:35:40.876559 7fcdffcdb700 10 log_queue is 1 last_log 56 sent 55 num 1 unsent 1 sending 1 -19480> 2013-04-03 10:35:40.876561 7fcdffcdb700 10 will send 2013-04-03 10:35:40.876558 mon.0 10.252.24.25:6789/0 56 : [INF] pgmap v28994: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 714B/s wr, 0op/s -19479> 2013-04-03 10:35:40.876567 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287c200 -19478> 2013-04-03 10:35:40.876613 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos -19477> 2013-04-03 10:35:40.876632 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos version 393999 summary v 393999 -19476> 2013-04-03 10:35:40.876650 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19475> 2013-04-03 10:35:40.876671 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287c200 con 0x1bac160 -19474> 2013-04-03 10:35:40.876692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos -19473> 2013-04-03 10:35:40.876707 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos version 393999 summary v 393999 -19472> 2013-04-03 10:35:40.876717 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19471> 2013-04-03 10:35:40.876729 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 preprocess_log log(1 entries) v1 from mon.0 -19470> 2013-04-03 10:35:40.876746 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19469> 2013-04-03 10:35:40.876758 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 prepare_log log(1 entries) v1 from mon.0 -19468> 2013-04-03 10:35:40.876768 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 logging 2013-04-03 10:35:40.876558 mon.0 10.252.24.25:6789/0 56 : [INF] pgmap v28994: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 714B/s wr, 0op/s -19467> 2013-04-03 10:35:40.887331 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113119 ==== paxos(lease_ack lc 427272 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (2787081935 0 0) 0x2981c80 con 0x208a840 -19466> 2013-04-03 10:35:40.887373 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos -19465> 2013-04-03 10:35:40.887390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 update_from_paxos version 393999 summary v 393999 -19464> 2013-04-03 10:35:40.887406 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19463> 2013-04-03 10:35:40.924462 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113120 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 33 v187) v2) to leader v1 ==== 560+0+0 (1801257011 0 0) 0x295ddc0 con 0x208a840 -19462> 2013-04-03 10:35:40.924475 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -19461> 2013-04-03 10:35:40.924483 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x295db00 from 10.252.24.88:6789/0 -19460> 2013-04-03 10:35:40.924513 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 33 v187) v2 from mds.0 10.252.24.25:6802/3144 -19459> 2013-04-03 10:35:40.924524 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 33 v187) v2 tid 17454) v2 -- ?+0 0x20adb40 con 0x208a840 -19458> 2013-04-03 10:35:41.378551 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 30 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21689 v187) v2 ==== 289+0+0 (3969305701 0 0) 0x2145000 con 0x200d420 -19457> 2013-04-03 10:35:41.378619 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21689 v187) v2 from mds.? 10.252.24.88:6800/17213 -19456> 2013-04-03 10:35:41.378635 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21689 v187) v2 -- ?+0 0x20e3dc0 con 0x200d420 -19455> 2013-04-03 10:35:41.875125 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v393999 encode_pending v394000 -19454> 2013-04-03 10:35:41.875154 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427258..427272) queue_proposal bl 442 bytes; ctx = 0x1b91a50 -19453> 2013-04-03 10:35:41.875162 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427258..427272) propose_queued 427273 442 bytes -19452> 2013-04-03 10:35:41.876339 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427272 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2981a00 -19451> 2013-04-03 10:35:41.913691 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113121 ==== paxos(accept lc 427272 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1820462242 0 0) 0x2981a00 con 0x208a840 -19450> 2013-04-03 10:35:41.913984 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427273 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2981c80 -19449> 2013-04-03 10:35:41.914018 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427273 fc 427258 pn 0 opn 0) v3 -- ?+0 0x2909a00 -19448> 2013-04-03 10:35:41.914056 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos -19447> 2013-04-03 10:35:41.914083 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 393999 -19446> 2013-04-03 10:35:41.914098 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos latest full 393798 -19445> 2013-04-03 10:35:41.914117 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos applying incremental log 394000 2013-04-03 10:35:40.876558 mon.0 10.252.24.25:6789/0 56 : [INF] pgmap v28994: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 714B/s wr, 0op/s -19444> 2013-04-03 10:35:41.914166 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 check_subs -19443> 2013-04-03 10:35:41.914197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 create_pending v 394001 -19442> 2013-04-03 10:35:41.914220 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 _updated_log for mon.0 10.252.24.25:6789/0 -19441> 2013-04-03 10:35:41.914233 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 56) v1 -- ?+0 0x28a5540 con 0x1bac160 -19440> 2013-04-03 10:35:41.914283 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos -19439> 2013-04-03 10:35:41.914299 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 -19438> 2013-04-03 10:35:41.914317 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19437> 2013-04-03 10:35:41.914339 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 56) v1 ==== 0+0+0 (0 0 0) 0x28a5540 con 0x1bac160 -19436> 2013-04-03 10:35:41.914347 7fcdffcdb700 10 handle_log_ack log(last 56) v1 -19435> 2013-04-03 10:35:41.914351 7fcdffcdb700 10 logged 2013-04-03 10:35:40.876558 mon.0 10.252.24.25:6789/0 56 : [INF] pgmap v28994: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 714B/s wr, 0op/s -19434> 2013-04-03 10:35:41.939570 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113122 ==== paxos(lease_ack lc 427273 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (698701413 0 0) 0x2909a00 con 0x208a840 -19433> 2013-04-03 10:35:41.939612 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos -19432> 2013-04-03 10:35:41.939629 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 -19431> 2013-04-03 10:35:41.939646 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19430> 2013-04-03 10:35:43.328218 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 check_down_pgs -19429> 2013-04-03 10:35:43.328250 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 v28994: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 714B/s wr, 0op/s -19428> 2013-04-03 10:35:43.328277 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -19427> 2013-04-03 10:35:43.328300 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in -19426> 2013-04-03 10:35:43.328338 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -19425> 2013-04-03 10:35:43.328369 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos -19424> 2013-04-03 10:35:43.328388 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 -19423> 2013-04-03 10:35:43.328401 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 log -19422> 2013-04-03 10:35:43.328418 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19421> 2013-04-03 10:35:43.328437 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -19420> 2013-04-03 10:35:43.578735 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 31 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140319 v187) v2 ==== 289+0+0 (2671704866 0 0) 0x20e4340 con 0x2118420 -19419> 2013-04-03 10:35:43.578781 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140319 v187) v2 from mds.? 10.252.58.31:6800/10908 -19418> 2013-04-03 10:35:43.578793 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140319 v187) v2 -- ?+0 0x2145000 con 0x2118420 -19417> 2013-04-03 10:35:44.914094 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427273 fc 427258 pn 0 opn 0) v3 -- ?+0 0x2981780 -19416> 2013-04-03 10:35:44.914876 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113123 ==== paxos(lease_ack lc 427273 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (2105500696 0 0) 0x2981780 con 0x208a840 -19415> 2013-04-03 10:35:44.914940 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos -19414> 2013-04-03 10:35:44.914961 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 -19413> 2013-04-03 10:35:44.914978 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19412> 2013-04-03 10:35:44.924520 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113124 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 34 v187) v2) to leader v1 ==== 560+0+0 (3903844434 0 0) 0x295e080 con 0x208a840 -19411> 2013-04-03 10:35:44.924533 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -19410> 2013-04-03 10:35:44.924542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x295d840 from 10.252.24.88:6789/0 -19409> 2013-04-03 10:35:44.924571 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 34 v187) v2 from mds.0 10.252.24.25:6802/3144 -19408> 2013-04-03 10:35:44.924584 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 34 v187) v2 tid 17455) v2 -- ?+0 0x24f8900 con 0x208a840 -19407> 2013-04-03 10:35:45.378707 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 31 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21690 v187) v2 ==== 289+0+0 (2300906978 0 0) 0x21452c0 con 0x200d420 -19406> 2013-04-03 10:35:45.378780 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21690 v187) v2 from mds.? 10.252.24.88:6800/17213 -19405> 2013-04-03 10:35:45.379059 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21690 v187) v2 -- ?+0 0x2991000 con 0x200d420 -19404> 2013-04-03 10:35:47.578859 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 32 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140320 v187) v2 ==== 289+0+0 (788227102 0 0) 0x273b080 con 0x2118420 -19403> 2013-04-03 10:35:47.578931 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140320 v187) v2 from mds.? 10.252.58.31:6800/10908 -19402> 2013-04-03 10:35:47.578948 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140320 v187) v2 -- ?+0 0x295e340 con 0x2118420 -19401> 2013-04-03 10:35:47.914164 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427273 fc 427258 pn 0 opn 0) v3 -- ?+0 0x2981500 -19400> 2013-04-03 10:35:47.914952 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113125 ==== paxos(lease_ack lc 427273 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (415499268 0 0) 0x2981500 con 0x208a840 -19399> 2013-04-03 10:35:47.915015 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos -19398> 2013-04-03 10:35:47.915036 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 -19397> 2013-04-03 10:35:47.915053 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19396> 2013-04-03 10:35:48.328583 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 v28994: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 714B/s wr, 0op/s -19395> 2013-04-03 10:35:48.328620 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -19394> 2013-04-03 10:35:48.328643 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in -19393> 2013-04-03 10:35:48.328687 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -19392> 2013-04-03 10:35:48.328718 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos -19391> 2013-04-03 10:35:48.328737 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 -19390> 2013-04-03 10:35:48.328747 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 log -19389> 2013-04-03 10:35:48.328768 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19388> 2013-04-03 10:35:48.328787 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -19387> 2013-04-03 10:35:48.484342 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113126 ==== forward(pg_stats(1 pgs tid 6 v 602) v1) to leader v1 ==== 757+0+0 (1400639085 0 0) 0x20e4340 con 0x208a840 -19386> 2013-04-03 10:35:48.484358 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 -19385> 2013-04-03 10:35:48.484368 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -19384> 2013-04-03 10:35:48.484400 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 preprocess_query pg_stats(1 pgs tid 6 v 602) v1 from osd.2 10.252.24.30:6800/1508 -19383> 2013-04-03 10:35:48.484421 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 prepare_update pg_stats(1 pgs tid 6 v 602) v1 from osd.2 10.252.24.30:6800/1508 -19382> 2013-04-03 10:35:48.484428 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 prepare_pg_stats pg_stats(1 pgs tid 6 v 602) v1 from osd.2 -19381> 2013-04-03 10:35:48.484434 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -19380> 2013-04-03 10:35:48.534614 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 encode_pending v 28995 -19379> 2013-04-03 10:35:48.534787 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427258..427273) queue_proposal bl 654 bytes; ctx = 0x1b91cf0 -19378> 2013-04-03 10:35:48.534799 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427258..427273) propose_queued 427274 654 bytes -19377> 2013-04-03 10:35:48.535853 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427273 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2981280 -19376> 2013-04-03 10:35:48.571829 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113127 ==== paxos(accept lc 427273 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2620676502 0 0) 0x2981280 con 0x208a840 -19375> 2013-04-03 10:35:48.572143 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427274 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2981500 -19374> 2013-04-03 10:35:48.572168 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427274 fc 427258 pn 0 opn 0) v3 -- ?+0 0x2981780 -19373> 2013-04-03 10:35:48.572244 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28994 update_from_paxos applying incremental 28995 -19372> 2013-04-03 10:35:48.572262 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 v28995: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1951B/s wr, 0op/s -19371> 2013-04-03 10:35:48.572281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 send_pg_creates to 0 pgs -19370> 2013-04-03 10:35:48.572286 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 update_logger -19369> 2013-04-03 10:35:48.572296 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 create_pending v 28996 -19368> 2013-04-03 10:35:48.572305 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 _updated_stats for osd.2 10.252.24.30:6800/1508 -19367> 2013-04-03 10:35:48.572312 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 6) v1 tid 17456) v2 -- ?+0 0x20adb40 con 0x208a840 -19366> 2013-04-03 10:35:48.572322 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 check_osd_map already seen 602 >= 602 -19365> 2013-04-03 10:35:48.572325 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 update_logger -19364> 2013-04-03 10:35:48.572336 7fcdffcdb700 0 log [INF] : pgmap v28995: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1951B/s wr, 0op/s -19363> 2013-04-03 10:35:48.572337 7fcdffcdb700 10 send_log to self -19362> 2013-04-03 10:35:48.572338 7fcdffcdb700 10 log_queue is 1 last_log 57 sent 56 num 1 unsent 1 sending 1 -19361> 2013-04-03 10:35:48.572340 7fcdffcdb700 10 will send 2013-04-03 10:35:48.572337 mon.0 10.252.24.25:6789/0 57 : [INF] pgmap v28995: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1951B/s wr, 0op/s -19360> 2013-04-03 10:35:48.572346 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287c200 -19359> 2013-04-03 10:35:48.572390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos -19358> 2013-04-03 10:35:48.572408 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 -19357> 2013-04-03 10:35:48.572425 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19356> 2013-04-03 10:35:48.572445 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287c200 con 0x1bac160 -19355> 2013-04-03 10:35:48.572467 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos -19354> 2013-04-03 10:35:48.572482 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 -19353> 2013-04-03 10:35:48.572494 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19352> 2013-04-03 10:35:48.572506 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 preprocess_log log(1 entries) v1 from mon.0 -19351> 2013-04-03 10:35:48.572528 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19350> 2013-04-03 10:35:48.572541 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 prepare_log log(1 entries) v1 from mon.0 -19349> 2013-04-03 10:35:48.572553 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 logging 2013-04-03 10:35:48.572337 mon.0 10.252.24.25:6789/0 57 : [INF] pgmap v28995: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1951B/s wr, 0op/s -19348> 2013-04-03 10:35:48.585514 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113128 ==== paxos(lease_ack lc 427274 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (3965776477 0 0) 0x2981780 con 0x208a840 -19347> 2013-04-03 10:35:48.585552 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos -19346> 2013-04-03 10:35:48.585569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 update_from_paxos version 394000 summary v 394000 -19345> 2013-04-03 10:35:48.585585 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19344> 2013-04-03 10:35:48.924769 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113129 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 35 v187) v2) to leader v1 ==== 560+0+0 (3186117544 0 0) 0x295d580 con 0x208a840 -19343> 2013-04-03 10:35:48.924791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -19342> 2013-04-03 10:35:48.924801 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x295d2c0 from 10.252.24.88:6789/0 -19341> 2013-04-03 10:35:48.924851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 35 v187) v2 from mds.0 10.252.24.25:6802/3144 -19340> 2013-04-03 10:35:48.924863 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 35 v187) v2 tid 17457) v2 -- ?+0 0x24f8900 con 0x208a840 -19339> 2013-04-03 10:35:49.378771 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 32 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21691 v187) v2 ==== 289+0+0 (2857087647 0 0) 0x2996340 con 0x200d420 -19338> 2013-04-03 10:35:49.378846 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21691 v187) v2 from mds.? 10.252.24.88:6800/17213 -19337> 2013-04-03 10:35:49.378860 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21691 v187) v2 -- ?+0 0x295d580 con 0x200d420 -19336> 2013-04-03 10:35:49.572011 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394000 encode_pending v394001 -19335> 2013-04-03 10:35:49.572053 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427258..427274) queue_proposal bl 443 bytes; ctx = 0x1b91990 -19334> 2013-04-03 10:35:49.572061 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427258..427274) propose_queued 427275 443 bytes -19333> 2013-04-03 10:35:49.573048 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427274 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2981000 -19332> 2013-04-03 10:35:49.602323 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113130 ==== paxos(accept lc 427274 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (88648733 0 0) 0x2981000 con 0x208a840 -19331> 2013-04-03 10:35:49.602649 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427275 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2981780 -19330> 2013-04-03 10:35:49.602674 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427275 fc 427258 pn 0 opn 0) v3 -- ?+0 0x2981280 -19329> 2013-04-03 10:35:49.602719 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos -19328> 2013-04-03 10:35:49.602756 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394000 -19327> 2013-04-03 10:35:49.602770 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos latest full 393798 -19326> 2013-04-03 10:35:49.602789 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos applying incremental log 394001 2013-04-03 10:35:48.572337 mon.0 10.252.24.25:6789/0 57 : [INF] pgmap v28995: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1951B/s wr, 0op/s -19325> 2013-04-03 10:35:49.602840 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 check_subs -19324> 2013-04-03 10:35:49.602872 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 create_pending v 394002 -19323> 2013-04-03 10:35:49.602894 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 _updated_log for mon.0 10.252.24.25:6789/0 -19322> 2013-04-03 10:35:49.602908 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 57) v1 -- ?+0 0x28a5380 con 0x1bac160 -19321> 2013-04-03 10:35:49.602958 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos -19320> 2013-04-03 10:35:49.602974 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394001 -19319> 2013-04-03 10:35:49.602992 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19318> 2013-04-03 10:35:49.603015 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 57) v1 ==== 0+0+0 (0 0 0) 0x28a5380 con 0x1bac160 -19317> 2013-04-03 10:35:49.603023 7fcdffcdb700 10 handle_log_ack log(last 57) v1 -19316> 2013-04-03 10:35:49.603025 7fcdffcdb700 10 logged 2013-04-03 10:35:48.572337 mon.0 10.252.24.25:6789/0 57 : [INF] pgmap v28995: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1951B/s wr, 0op/s -19315> 2013-04-03 10:35:49.628402 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113131 ==== paxos(lease_ack lc 427275 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (3276791632 0 0) 0x2981280 con 0x208a840 -19314> 2013-04-03 10:35:49.628484 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos -19313> 2013-04-03 10:35:49.628504 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394001 -19312> 2013-04-03 10:35:49.628521 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19311> 2013-04-03 10:35:51.579008 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 33 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140321 v187) v2 ==== 289+0+0 (227924835 0 0) 0x273a2c0 con 0x2118420 -19310> 2013-04-03 10:35:51.579079 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140321 v187) v2 from mds.? 10.252.58.31:6800/10908 -19309> 2013-04-03 10:35:51.579093 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140321 v187) v2 -- ?+0 0x2996340 con 0x2118420 -19308> 2013-04-03 10:35:52.602756 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427275 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29a1c80 -19307> 2013-04-03 10:35:52.603462 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113132 ==== paxos(lease_ack lc 427275 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (3969056678 0 0) 0x29a1c80 con 0x208a840 -19306> 2013-04-03 10:35:52.603534 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos -19305> 2013-04-03 10:35:52.603555 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394001 -19304> 2013-04-03 10:35:52.603572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19303> 2013-04-03 10:35:52.924941 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113133 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 36 v187) v2) to leader v1 ==== 560+0+0 (3064464280 0 0) 0x20e4340 con 0x208a840 -19302> 2013-04-03 10:35:52.924962 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -19301> 2013-04-03 10:35:52.924973 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x295d000 from 10.252.24.88:6789/0 -19300> 2013-04-03 10:35:52.925017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 36 v187) v2 from mds.0 10.252.24.25:6802/3144 -19299> 2013-04-03 10:35:52.925029 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 36 v187) v2 tid 17458) v2 -- ?+0 0x20adb40 con 0x208a840 -19298> 2013-04-03 10:35:53.328922 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 check_down_pgs -19297> 2013-04-03 10:35:53.328952 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 v28995: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1951B/s wr, 0op/s -19296> 2013-04-03 10:35:53.328979 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -19295> 2013-04-03 10:35:53.329004 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in -19294> 2013-04-03 10:35:53.329041 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -19293> 2013-04-03 10:35:53.329346 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos -19292> 2013-04-03 10:35:53.329370 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394001 -19291> 2013-04-03 10:35:53.329380 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 log -19290> 2013-04-03 10:35:53.329397 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19289> 2013-04-03 10:35:53.329415 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -19288> 2013-04-03 10:35:53.378893 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 33 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21692 v187) v2 ==== 289+0+0 (1111658220 0 0) 0x2996080 con 0x200d420 -19287> 2013-04-03 10:35:53.378944 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21692 v187) v2 from mds.? 10.252.24.88:6800/17213 -19286> 2013-04-03 10:35:53.378956 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21692 v187) v2 -- ?+0 0x20e4340 con 0x200d420 -19285> 2013-04-03 10:35:54.564109 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113134 ==== forward(pg_stats(1 pgs tid 14 v 602) v1) to leader v1 ==== 757+0+0 (3911307504 0 0) 0x273a2c0 con 0x208a840 -19284> 2013-04-03 10:35:54.564129 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -19283> 2013-04-03 10:35:54.564139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -19282> 2013-04-03 10:35:54.564190 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 preprocess_query pg_stats(1 pgs tid 14 v 602) v1 from osd.1 10.252.24.25:6805/3572 -19281> 2013-04-03 10:35:54.564218 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 prepare_update pg_stats(1 pgs tid 14 v 602) v1 from osd.1 10.252.24.25:6805/3572 -19280> 2013-04-03 10:35:54.564224 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 prepare_pg_stats pg_stats(1 pgs tid 14 v 602) v1 from osd.1 -19279> 2013-04-03 10:35:54.564231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -19278> 2013-04-03 10:35:54.614410 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 encode_pending v 28996 -19277> 2013-04-03 10:35:54.614450 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427258..427275) queue_proposal bl 654 bytes; ctx = 0x1b918a0 -19276> 2013-04-03 10:35:54.614460 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427258..427275) propose_queued 427276 654 bytes -19275> 2013-04-03 10:35:54.616010 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427275 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1a00 -19274> 2013-04-03 10:35:54.644444 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113135 ==== paxos(accept lc 427275 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (962113235 0 0) 0x29a1a00 con 0x208a840 -19273> 2013-04-03 10:35:54.644782 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427276 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1c80 -19272> 2013-04-03 10:35:54.644809 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427276 fc 427258 pn 0 opn 0) v3 -- ?+0 0x2981280 -19271> 2013-04-03 10:35:54.644882 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28995 update_from_paxos applying incremental 28996 -19270> 2013-04-03 10:35:54.644898 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 v28996: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s -19269> 2013-04-03 10:35:54.644919 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 send_pg_creates to 0 pgs -19268> 2013-04-03 10:35:54.644922 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 update_logger -19267> 2013-04-03 10:35:54.644932 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 create_pending v 28997 -19266> 2013-04-03 10:35:54.644941 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 _updated_stats for osd.1 10.252.24.25:6805/3572 -19265> 2013-04-03 10:35:54.644947 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 14) v1 tid 17459) v2 -- ?+0 0x24f8900 con 0x208a840 -19264> 2013-04-03 10:35:54.644956 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 check_osd_map already seen 602 >= 602 -19263> 2013-04-03 10:35:54.644960 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 update_logger -19262> 2013-04-03 10:35:54.644971 7fcdffcdb700 0 log [INF] : pgmap v28996: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s -19261> 2013-04-03 10:35:54.644973 7fcdffcdb700 10 send_log to self -19260> 2013-04-03 10:35:54.644973 7fcdffcdb700 10 log_queue is 1 last_log 58 sent 57 num 1 unsent 1 sending 1 -19259> 2013-04-03 10:35:54.644975 7fcdffcdb700 10 will send 2013-04-03 10:35:54.644972 mon.0 10.252.24.25:6789/0 58 : [INF] pgmap v28996: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s -19258> 2013-04-03 10:35:54.644994 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287c200 -19257> 2013-04-03 10:35:54.645043 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos -19256> 2013-04-03 10:35:54.645063 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394001 -19255> 2013-04-03 10:35:54.645082 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19254> 2013-04-03 10:35:54.645103 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287c200 con 0x1bac160 -19253> 2013-04-03 10:35:54.645125 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos -19252> 2013-04-03 10:35:54.645140 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394001 -19251> 2013-04-03 10:35:54.645149 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19250> 2013-04-03 10:35:54.645160 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 preprocess_log log(1 entries) v1 from mon.0 -19249> 2013-04-03 10:35:54.645181 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19248> 2013-04-03 10:35:54.645194 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 prepare_log log(1 entries) v1 from mon.0 -19247> 2013-04-03 10:35:54.645205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 logging 2013-04-03 10:35:54.644972 mon.0 10.252.24.25:6789/0 58 : [INF] pgmap v28996: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s -19246> 2013-04-03 10:35:54.656874 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113136 ==== paxos(lease_ack lc 427276 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (2117839949 0 0) 0x2981280 con 0x208a840 -19245> 2013-04-03 10:35:54.656910 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos -19244> 2013-04-03 10:35:54.656926 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 update_from_paxos version 394001 summary v 394001 -19243> 2013-04-03 10:35:54.656941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19242> 2013-04-03 10:35:55.579129 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 34 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140322 v187) v2 ==== 289+0+0 (1747327716 0 0) 0x273a000 con 0x2118420 -19241> 2013-04-03 10:35:55.579202 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140322 v187) v2 from mds.? 10.252.58.31:6800/10908 -19240> 2013-04-03 10:35:55.579215 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140322 v187) v2 -- ?+0 0x273a2c0 con 0x2118420 -19239> 2013-04-03 10:35:55.644628 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394001 encode_pending v394002 -19238> 2013-04-03 10:35:55.644656 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427258..427276) queue_proposal bl 443 bytes; ctx = 0x1b91a70 -19237> 2013-04-03 10:35:55.644663 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427258..427276) propose_queued 427277 443 bytes -19236> 2013-04-03 10:35:55.646237 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427276 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1780 -19235> 2013-04-03 10:35:55.673646 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113137 ==== paxos(accept lc 427276 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2471552910 0 0) 0x29a1780 con 0x208a840 -19234> 2013-04-03 10:35:55.673938 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427277 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2981280 -19233> 2013-04-03 10:35:55.673970 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427277 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29a1a00 -19232> 2013-04-03 10:35:55.674009 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos -19231> 2013-04-03 10:35:55.674035 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394001 -19230> 2013-04-03 10:35:55.674050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos latest full 393798 -19229> 2013-04-03 10:35:55.674069 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos applying incremental log 394002 2013-04-03 10:35:54.644972 mon.0 10.252.24.25:6789/0 58 : [INF] pgmap v28996: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s -19228> 2013-04-03 10:35:55.674114 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 check_subs -19227> 2013-04-03 10:35:55.674146 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 create_pending v 394003 -19226> 2013-04-03 10:35:55.674169 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 _updated_log for mon.0 10.252.24.25:6789/0 -19225> 2013-04-03 10:35:55.674182 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 58) v1 -- ?+0 0x28a51c0 con 0x1bac160 -19224> 2013-04-03 10:35:55.674230 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos -19223> 2013-04-03 10:35:55.674246 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 -19222> 2013-04-03 10:35:55.674263 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19221> 2013-04-03 10:35:55.674283 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 58) v1 ==== 0+0+0 (0 0 0) 0x28a51c0 con 0x1bac160 -19220> 2013-04-03 10:35:55.674292 7fcdffcdb700 10 handle_log_ack log(last 58) v1 -19219> 2013-04-03 10:35:55.674293 7fcdffcdb700 10 logged 2013-04-03 10:35:54.644972 mon.0 10.252.24.25:6789/0 58 : [INF] pgmap v28996: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s -19218> 2013-04-03 10:35:55.685916 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113138 ==== paxos(lease_ack lc 427277 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (149201370 0 0) 0x29a1a00 con 0x208a840 -19217> 2013-04-03 10:35:55.685953 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos -19216> 2013-04-03 10:35:55.685973 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 -19215> 2013-04-03 10:35:55.685988 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19214> 2013-04-03 10:35:56.925120 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113139 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 37 v187) v2) to leader v1 ==== 560+0+0 (1583371508 0 0) 0x2996b80 con 0x208a840 -19213> 2013-04-03 10:35:56.925142 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -19212> 2013-04-03 10:35:56.925153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29968c0 from 10.252.24.88:6789/0 -19211> 2013-04-03 10:35:56.925204 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 37 v187) v2 from mds.0 10.252.24.25:6802/3144 -19210> 2013-04-03 10:35:56.925216 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 37 v187) v2 tid 17460) v2 -- ?+0 0x20adb40 con 0x208a840 -19209> 2013-04-03 10:35:57.378971 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 34 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21693 v187) v2 ==== 289+0+0 (1630286225 0 0) 0x2995dc0 con 0x200d420 -19208> 2013-04-03 10:35:57.379051 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21693 v187) v2 from mds.? 10.252.24.88:6800/17213 -19207> 2013-04-03 10:35:57.379066 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21693 v187) v2 -- ?+0 0x2996b80 con 0x200d420 -19206> 2013-04-03 10:35:58.329557 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 check_down_pgs -19205> 2013-04-03 10:35:58.329593 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 v28996: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s -19204> 2013-04-03 10:35:58.329629 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -19203> 2013-04-03 10:35:58.329654 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in -19202> 2013-04-03 10:35:58.329694 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -19201> 2013-04-03 10:35:58.329725 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos -19200> 2013-04-03 10:35:58.329744 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 -19199> 2013-04-03 10:35:58.329754 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 log -19198> 2013-04-03 10:35:58.329771 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19197> 2013-04-03 10:35:58.329790 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -19196> 2013-04-03 10:35:58.674045 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427277 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29a1500 -19195> 2013-04-03 10:35:58.674859 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113140 ==== paxos(lease_ack lc 427277 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (3389767196 0 0) 0x29a1500 con 0x208a840 -19194> 2013-04-03 10:35:58.674923 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos -19193> 2013-04-03 10:35:58.674943 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 -19192> 2013-04-03 10:35:58.674960 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19191> 2013-04-03 10:35:59.579259 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 35 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140323 v187) v2 ==== 289+0+0 (1263052185 0 0) 0x273a580 con 0x2118420 -19190> 2013-04-03 10:35:59.579330 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140323 v187) v2 from mds.? 10.252.58.31:6800/10908 -19189> 2013-04-03 10:35:59.579344 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140323 v187) v2 -- ?+0 0x2995dc0 con 0x2118420 -19188> 2013-04-03 10:36:00.249168 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x29b4780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:32978/0 -19187> 2013-04-03 10:36:00.249202 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:32978/0) -19186> 2013-04-03 10:36:00.249215 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4780 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -19185> 2013-04-03 10:36:00.249418 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4780 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -19184> 2013-04-03 10:36:00.925150 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113141 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 38 v187) v2) to leader v1 ==== 560+0+0 (3717375125 0 0) 0x273a000 con 0x208a840 -19183> 2013-04-03 10:36:00.925170 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -19182> 2013-04-03 10:36:00.925182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2996600 from 10.252.24.88:6789/0 -19181> 2013-04-03 10:36:00.925232 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 38 v187) v2 from mds.0 10.252.24.25:6802/3144 -19180> 2013-04-03 10:36:00.925246 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 38 v187) v2 tid 17461) v2 -- ?+0 0x24f8900 con 0x208a840 -19179> 2013-04-03 10:36:01.379115 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 35 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21694 v187) v2 ==== 289+0+0 (77590550 0 0) 0x2995b00 con 0x200d420 -19178> 2013-04-03 10:36:01.379184 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21694 v187) v2 from mds.? 10.252.24.88:6800/17213 -19177> 2013-04-03 10:36:01.379197 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21694 v187) v2 -- ?+0 0x273a000 con 0x200d420 -19176> 2013-04-03 10:36:01.674113 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427277 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29a1280 -19175> 2013-04-03 10:36:01.674840 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113142 ==== paxos(lease_ack lc 427277 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (4036068111 0 0) 0x29a1280 con 0x208a840 -19174> 2013-04-03 10:36:01.674893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos -19173> 2013-04-03 10:36:01.674913 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 -19172> 2013-04-03 10:36:01.674930 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19171> 2013-04-03 10:36:03.329938 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 v28996: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s -19170> 2013-04-03 10:36:03.329975 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -19169> 2013-04-03 10:36:03.329999 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in -19168> 2013-04-03 10:36:03.330042 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -19167> 2013-04-03 10:36:03.330077 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos -19166> 2013-04-03 10:36:03.330096 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 -19165> 2013-04-03 10:36:03.330106 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 log -19164> 2013-04-03 10:36:03.330123 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19163> 2013-04-03 10:36:03.330142 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -19162> 2013-04-03 10:36:03.486301 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113143 ==== forward(pg_stats(1 pgs tid 7 v 602) v1) to leader v1 ==== 757+0+0 (1584844878 0 0) 0x273a580 con 0x208a840 -19161> 2013-04-03 10:36:03.486318 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 -19160> 2013-04-03 10:36:03.486327 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -19159> 2013-04-03 10:36:03.486362 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 preprocess_query pg_stats(1 pgs tid 7 v 602) v1 from osd.2 10.252.24.30:6800/1508 -19158> 2013-04-03 10:36:03.486385 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 prepare_update pg_stats(1 pgs tid 7 v 602) v1 from osd.2 10.252.24.30:6800/1508 -19157> 2013-04-03 10:36:03.486394 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 prepare_pg_stats pg_stats(1 pgs tid 7 v 602) v1 from osd.2 -19156> 2013-04-03 10:36:03.486400 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -19155> 2013-04-03 10:36:03.537065 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 encode_pending v 28997 -19154> 2013-04-03 10:36:03.537103 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427258..427277) queue_proposal bl 654 bytes; ctx = 0x1b91cd0 -19153> 2013-04-03 10:36:03.537112 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427258..427277) propose_queued 427278 654 bytes -19152> 2013-04-03 10:36:03.538050 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427277 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1000 -19151> 2013-04-03 10:36:03.579400 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 36 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140324 v187) v2 ==== 289+0+0 (2738985450 0 0) 0x273a840 con 0x2118420 -19150> 2013-04-03 10:36:03.579449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140324 v187) v2 from mds.? 10.252.58.31:6800/10908 -19149> 2013-04-03 10:36:03.579460 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140324 v187) v2 -- ?+0 0x273a580 con 0x2118420 -19148> 2013-04-03 10:36:03.587031 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113144 ==== paxos(accept lc 427277 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2054732158 0 0) 0x29a1000 con 0x208a840 -19147> 2013-04-03 10:36:03.587334 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427278 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1280 -19146> 2013-04-03 10:36:03.587366 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427278 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29a1500 -19145> 2013-04-03 10:36:03.587434 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28996 update_from_paxos applying incremental 28997 -19144> 2013-04-03 10:36:03.587450 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 v28997: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19143> 2013-04-03 10:36:03.587469 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 send_pg_creates to 0 pgs -19142> 2013-04-03 10:36:03.587473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 update_logger -19141> 2013-04-03 10:36:03.587483 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 create_pending v 28998 -19140> 2013-04-03 10:36:03.587493 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 _updated_stats for osd.2 10.252.24.30:6800/1508 -19139> 2013-04-03 10:36:03.587499 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 7) v1 tid 17462) v2 -- ?+0 0x20adb40 con 0x208a840 -19138> 2013-04-03 10:36:03.587510 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 check_osd_map already seen 602 >= 602 -19137> 2013-04-03 10:36:03.587514 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 update_logger -19136> 2013-04-03 10:36:03.587524 7fcdffcdb700 0 log [INF] : pgmap v28997: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19135> 2013-04-03 10:36:03.587526 7fcdffcdb700 10 send_log to self -19134> 2013-04-03 10:36:03.587531 7fcdffcdb700 10 log_queue is 1 last_log 59 sent 58 num 1 unsent 1 sending 1 -19133> 2013-04-03 10:36:03.587533 7fcdffcdb700 10 will send 2013-04-03 10:36:03.587526 mon.0 10.252.24.25:6789/0 59 : [INF] pgmap v28997: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19132> 2013-04-03 10:36:03.587541 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287c200 -19131> 2013-04-03 10:36:03.587588 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos -19130> 2013-04-03 10:36:03.587605 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 -19129> 2013-04-03 10:36:03.587623 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19128> 2013-04-03 10:36:03.587644 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287c200 con 0x1bac160 -19127> 2013-04-03 10:36:03.587665 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos -19126> 2013-04-03 10:36:03.587680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 -19125> 2013-04-03 10:36:03.587690 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19124> 2013-04-03 10:36:03.587702 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 preprocess_log log(1 entries) v1 from mon.0 -19123> 2013-04-03 10:36:03.587724 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19122> 2013-04-03 10:36:03.587737 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 prepare_log log(1 entries) v1 from mon.0 -19121> 2013-04-03 10:36:03.587748 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 logging 2013-04-03 10:36:03.587526 mon.0 10.252.24.25:6789/0 59 : [INF] pgmap v28997: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19120> 2013-04-03 10:36:03.600282 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113145 ==== paxos(lease_ack lc 427278 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (205718809 0 0) 0x29a1500 con 0x208a840 -19119> 2013-04-03 10:36:03.600318 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos -19118> 2013-04-03 10:36:03.600335 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 update_from_paxos version 394002 summary v 394002 -19117> 2013-04-03 10:36:03.600351 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19116> 2013-04-03 10:36:04.587228 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394002 encode_pending v394003 -19115> 2013-04-03 10:36:04.587257 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427258..427278) queue_proposal bl 443 bytes; ctx = 0x1b91d60 -19114> 2013-04-03 10:36:04.587265 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427258..427278) propose_queued 427279 443 bytes -19113> 2013-04-03 10:36:04.588322 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427278 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4c80 -19112> 2013-04-03 10:36:04.615388 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113146 ==== paxos(accept lc 427278 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1646993328 0 0) 0x29b4c80 con 0x208a840 -19111> 2013-04-03 10:36:04.616218 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427279 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1500 -19110> 2013-04-03 10:36:04.616251 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427279 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29a1000 -19109> 2013-04-03 10:36:04.616290 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos -19108> 2013-04-03 10:36:04.616314 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos version 394003 summary v 394002 -19107> 2013-04-03 10:36:04.616331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos latest full 393798 -19106> 2013-04-03 10:36:04.616350 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos applying incremental log 394003 2013-04-03 10:36:03.587526 mon.0 10.252.24.25:6789/0 59 : [INF] pgmap v28997: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19105> 2013-04-03 10:36:04.616395 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 check_subs -19104> 2013-04-03 10:36:04.616426 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 create_pending v 394004 -19103> 2013-04-03 10:36:04.616449 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 _updated_log for mon.0 10.252.24.25:6789/0 -19102> 2013-04-03 10:36:04.616462 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 59) v1 -- ?+0 0x2294e00 con 0x1bac160 -19101> 2013-04-03 10:36:04.616512 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos -19100> 2013-04-03 10:36:04.616528 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos version 394003 summary v 394003 -19099> 2013-04-03 10:36:04.616544 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19098> 2013-04-03 10:36:04.616693 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 59) v1 ==== 0+0+0 (0 0 0) 0x2294e00 con 0x1bac160 -19097> 2013-04-03 10:36:04.616703 7fcdffcdb700 10 handle_log_ack log(last 59) v1 -19096> 2013-04-03 10:36:04.616704 7fcdffcdb700 10 logged 2013-04-03 10:36:03.587526 mon.0 10.252.24.25:6789/0 59 : [INF] pgmap v28997: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19095> 2013-04-03 10:36:04.628627 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113147 ==== paxos(lease_ack lc 427279 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (1681515472 0 0) 0x29a1000 con 0x208a840 -19094> 2013-04-03 10:36:04.628663 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos -19093> 2013-04-03 10:36:04.628680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos version 394003 summary v 394003 -19092> 2013-04-03 10:36:04.628699 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19091> 2013-04-03 10:36:04.925324 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113148 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 39 v187) v2) to leader v1 ==== 560+0+0 (15180224 0 0) 0x29c5b80 con 0x208a840 -19090> 2013-04-03 10:36:04.925347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -19089> 2013-04-03 10:36:04.925358 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29c58c0 from 10.252.24.88:6789/0 -19088> 2013-04-03 10:36:04.925409 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 39 v187) v2 from mds.0 10.252.24.25:6802/3144 -19087> 2013-04-03 10:36:04.925421 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 39 v187) v2 tid 17463) v2 -- ?+0 0x24f8900 con 0x208a840 -19086> 2013-04-03 10:36:05.379294 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 36 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21695 v187) v2 ==== 289+0+0 (670125931 0 0) 0x2995840 con 0x200d420 -19085> 2013-04-03 10:36:05.379369 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21695 v187) v2 from mds.? 10.252.24.88:6800/17213 -19084> 2013-04-03 10:36:05.379382 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21695 v187) v2 -- ?+0 0x29c5b80 con 0x200d420 -19083> 2013-04-03 10:36:06.473510 7fcdfe8d5700 2 -- 10.252.24.25:6789/0 >> 10.252.24.30:6789/0 pipe(0x26dc500 sd=24 :6789 s=4 pgs=0 cs=0 l=0).fault 0: Success -19082> 2013-04-03 10:36:07.579508 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 37 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140325 v187) v2 ==== 289+0+0 (2150573719 0 0) 0x273b8c0 con 0x2118420 -19081> 2013-04-03 10:36:07.579578 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140325 v187) v2 from mds.? 10.252.58.31:6800/10908 -19080> 2013-04-03 10:36:07.579591 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140325 v187) v2 -- ?+0 0x2995840 con 0x2118420 -19079> 2013-04-03 10:36:07.616319 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427279 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29b4a00 -19078> 2013-04-03 10:36:07.617160 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113149 ==== paxos(lease_ack lc 427279 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (1035434345 0 0) 0x29b4a00 con 0x208a840 -19077> 2013-04-03 10:36:07.617213 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos -19076> 2013-04-03 10:36:07.617234 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos version 394003 summary v 394003 -19075> 2013-04-03 10:36:07.617252 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19074> 2013-04-03 10:36:08.330291 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 check_down_pgs -19073> 2013-04-03 10:36:08.330322 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v28997 v28997: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19072> 2013-04-03 10:36:08.330351 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -19071> 2013-04-03 10:36:08.330376 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in -19070> 2013-04-03 10:36:08.330413 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -19069> 2013-04-03 10:36:08.330448 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos -19068> 2013-04-03 10:36:08.330467 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 update_from_paxos version 394003 summary v 394003 -19067> 2013-04-03 10:36:08.330477 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394003 log -19066> 2013-04-03 10:36:08.330494 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -19065> 2013-04-03 10:36:08.330513 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -19064> 2013-04-03 10:36:08.627484 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) -19063> 2013-04-03 10:36:08.627511 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader -19062> 2013-04-03 10:36:08.925472 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113150 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 40 v187) v2) to leader v1 ==== 560+0+0 (1951681489 0 0) 0x273a840 con 0x208a840 -19061> 2013-04-03 10:36:08.925494 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -19060> 2013-04-03 10:36:08.925504 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c5600 from 10.252.24.88:6789/0 -19059> 2013-04-03 10:36:08.925557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 40 v187) v2 from mds.0 10.252.24.25:6802/3144 -19058> 2013-04-03 10:36:08.925575 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 40 v187) v2 tid 17464) v2 -- ?+0 0x20adb40 con 0x208a840 -19057> 2013-04-03 10:36:09.070456 7fcdfe8d5700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x29b4500 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept sd=24 10.252.24.30:45363/0 -19056> 2013-04-03 10:36:09.070840 7fcdfe8d5700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.30:6789/0 mon protocol 2 -19055> 2013-04-03 10:36:09.070944 7fcdfe8d5700 0 -- 10.252.24.25:6789/0 >> 10.252.24.30:6789/0 pipe(0x29b4500 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state connecting -19054> 2013-04-03 10:36:09.087261 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113151 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (1960963407 0 0) 0x273b8c0 con 0x208a840 -19053> 2013-04-03 10:36:09.087280 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -19052> 2013-04-03 10:36:09.087287 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -19051> 2013-04-03 10:36:09.087292 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( start_reply flags( retry ) ) v1 -- ?+0 0x273a840 -19050> 2013-04-03 10:36:09.379349 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 37 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21696 v187) v2 ==== 289+0+0 (1660467359 0 0) 0x2995580 con 0x200d420 -19049> 2013-04-03 10:36:09.379416 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21696 v187) v2 from mds.? 10.252.24.88:6800/17213 -19048> 2013-04-03 10:36:09.379428 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21696 v187) v2 -- ?+0 0x273b8c0 con 0x200d420 -19047> 2013-04-03 10:36:09.565919 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113152 ==== forward(pg_stats(1 pgs tid 15 v 602) v1) to leader v1 ==== 757+0+0 (4289216560 0 0) 0x29c5340 con 0x208a840 -19046> 2013-04-03 10:36:09.565931 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -19045> 2013-04-03 10:36:09.565940 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -19044> 2013-04-03 10:36:09.565973 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28997 preprocess_query pg_stats(1 pgs tid 15 v 602) v1 from osd.1 10.252.24.25:6805/3572 -19043> 2013-04-03 10:36:09.565996 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28997 prepare_update pg_stats(1 pgs tid 15 v 602) v1 from osd.1 10.252.24.25:6805/3572 -19042> 2013-04-03 10:36:09.566001 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28997 prepare_pg_stats pg_stats(1 pgs tid 15 v 602) v1 from osd.1 -19041> 2013-04-03 10:36:09.566007 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28997 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -19040> 2013-04-03 10:36:09.616187 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28997 encode_pending v 28998 -19039> 2013-04-03 10:36:09.616217 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427279) queue_proposal bl 654 bytes; ctx = 0x1b91d10 -19038> 2013-04-03 10:36:09.616227 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427279) propose_queued 427280 654 bytes -19037> 2013-04-03 10:36:09.618034 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427279 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4780 -19036> 2013-04-03 10:36:09.648457 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113153 ==== paxos(accept lc 427279 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (95007105 0 0) 0x29b4780 con 0x208a840 -19035> 2013-04-03 10:36:09.648786 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427280 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4a00 -19034> 2013-04-03 10:36:09.648821 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427280 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29a1000 -19033> 2013-04-03 10:36:09.648889 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28997 update_from_paxos applying incremental 28998 -19032> 2013-04-03 10:36:09.648904 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 v28998: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19031> 2013-04-03 10:36:09.648923 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 send_pg_creates to 0 pgs -19030> 2013-04-03 10:36:09.648927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 update_logger -19029> 2013-04-03 10:36:09.648937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 create_pending v 28999 -19028> 2013-04-03 10:36:09.648945 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28998 _updated_stats for osd.1 10.252.24.25:6805/3572 -19027> 2013-04-03 10:36:09.648952 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 15) v1 tid 17465) v2 -- ?+0 0x24f8900 con 0x208a840 -19026> 2013-04-03 10:36:09.648961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 check_osd_map already seen 602 >= 602 -19025> 2013-04-03 10:36:09.648964 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 update_logger -19024> 2013-04-03 10:36:09.648974 7fcdffcdb700 0 log [INF] : pgmap v28998: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19023> 2013-04-03 10:36:09.648976 7fcdffcdb700 10 send_log to self -19022> 2013-04-03 10:36:09.648976 7fcdffcdb700 10 log_queue is 1 last_log 60 sent 59 num 1 unsent 1 sending 1 -19021> 2013-04-03 10:36:09.648978 7fcdffcdb700 10 will send 2013-04-03 10:36:09.648975 mon.0 10.252.24.25:6789/0 60 : [INF] pgmap v28998: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19020> 2013-04-03 10:36:09.648985 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287c200 -19019> 2013-04-03 10:36:09.649031 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 update_from_paxos -19018> 2013-04-03 10:36:09.649050 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 update_from_paxos version 394003 summary v 394003 -19017> 2013-04-03 10:36:09.649069 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19016> 2013-04-03 10:36:09.649120 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287c200 con 0x1bac160 -19015> 2013-04-03 10:36:09.649143 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 update_from_paxos -19014> 2013-04-03 10:36:09.649158 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 update_from_paxos version 394003 summary v 394003 -19013> 2013-04-03 10:36:09.649166 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19012> 2013-04-03 10:36:09.649178 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 preprocess_log log(1 entries) v1 from mon.0 -19011> 2013-04-03 10:36:09.649200 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -19010> 2013-04-03 10:36:09.649212 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 prepare_log log(1 entries) v1 from mon.0 -19009> 2013-04-03 10:36:09.649223 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 logging 2013-04-03 10:36:09.648975 mon.0 10.252.24.25:6789/0 60 : [INF] pgmap v28998: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -19008> 2013-04-03 10:36:09.660396 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113154 ==== paxos(lease_ack lc 427280 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (3320997639 0 0) 0x29a1000 con 0x208a840 -19007> 2013-04-03 10:36:09.660434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 update_from_paxos -19006> 2013-04-03 10:36:09.660449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394003 update_from_paxos version 394003 summary v 394003 -19005> 2013-04-03 10:36:09.660464 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -19004> 2013-04-03 10:36:09.732858 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113155 ==== forward(pg_stats(1 pgs tid 30438 v 602) v1) to leader v1 ==== 757+0+0 (2672988918 0 0) 0x29c5080 con 0x208a840 -19003> 2013-04-03 10:36:09.732867 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -19002> 2013-04-03 10:36:09.732875 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -19001> 2013-04-03 10:36:09.732899 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 preprocess_query pg_stats(1 pgs tid 30438 v 602) v1 from osd.3 10.252.24.88:6801/17538 -19000> 2013-04-03 10:36:09.732918 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 prepare_update pg_stats(1 pgs tid 30438 v 602) v1 from osd.3 10.252.24.88:6801/17538 -18999> 2013-04-03 10:36:09.732924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 prepare_pg_stats pg_stats(1 pgs tid 30438 v 602) v1 from osd.3 -18998> 2013-04-03 10:36:09.732930 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28998 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -18997> 2013-04-03 10:36:10.648653 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28998 encode_pending v 28999 -18996> 2013-04-03 10:36:10.648688 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427280) queue_proposal bl 654 bytes; ctx = 0x1b91c40 -18995> 2013-04-03 10:36:10.648695 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427280) propose_queued 427281 654 bytes -18994> 2013-04-03 10:36:10.650358 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427280 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc500 -18993> 2013-04-03 10:36:10.650467 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394003 encode_pending v394004 -18992> 2013-04-03 10:36:10.650904 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427258..427280) queue_proposal bl 443 bytes; ctx = 0x1b91cc0 -18991> 2013-04-03 10:36:10.650914 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427258..427280) propose_new_value not active; proposal queued -18990> 2013-04-03 10:36:10.676279 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113156 ==== paxos(accept lc 427280 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2380182875 0 0) 0x26dc500 con 0x208a840 -18989> 2013-04-03 10:36:10.676613 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427281 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1000 -18988> 2013-04-03 10:36:10.676646 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427281 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29b4780 -18987> 2013-04-03 10:36:10.676715 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28998 update_from_paxos applying incremental 28999 -18986> 2013-04-03 10:36:10.676735 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 v28999: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s -18985> 2013-04-03 10:36:10.676754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 send_pg_creates to 0 pgs -18984> 2013-04-03 10:36:10.676757 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 update_logger -18983> 2013-04-03 10:36:10.676766 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 create_pending v 29000 -18982> 2013-04-03 10:36:10.676775 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28999 _updated_stats for osd.3 10.252.24.88:6801/17538 -18981> 2013-04-03 10:36:10.676782 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30438) v1 tid 17466) v2 -- ?+0 0x20adb40 con 0x208a840 -18980> 2013-04-03 10:36:10.676793 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 check_osd_map already seen 602 >= 602 -18979> 2013-04-03 10:36:10.676796 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 update_logger -18978> 2013-04-03 10:36:10.676806 7fcdffcdb700 0 log [INF] : pgmap v28999: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s -18977> 2013-04-03 10:36:10.676807 7fcdffcdb700 10 send_log to self -18976> 2013-04-03 10:36:10.676808 7fcdffcdb700 10 log_queue is 2 last_log 61 sent 60 num 2 unsent 1 sending 1 -18975> 2013-04-03 10:36:10.676810 7fcdffcdb700 10 will send 2013-04-03 10:36:09.648975 mon.0 10.252.24.25:6789/0 60 : [INF] pgmap v28998: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18974> 2013-04-03 10:36:10.676816 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2874800 -18973> 2013-04-03 10:36:10.676842 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427281) propose_queued 427282 443 bytes -18972> 2013-04-03 10:36:10.677151 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427281 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4c80 -18971> 2013-04-03 10:36:10.677186 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2874800 con 0x1bac160 -18970> 2013-04-03 10:36:10.689551 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113157 ==== paxos(lease_ack lc 427281 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (1097747691 0 0) 0x29b4c80 con 0x208a840 -18969> 2013-04-03 10:36:10.713377 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113158 ==== paxos(accept lc 427281 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2758451468 0 0) 0x29b4780 con 0x208a840 -18968> 2013-04-03 10:36:10.714434 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427282 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4c80 -18967> 2013-04-03 10:36:10.714466 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427282 fc 427258 pn 0 opn 0) v3 -- ?+0 0x26dc500 -18966> 2013-04-03 10:36:10.714503 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos -18965> 2013-04-03 10:36:10.714528 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394003 -18964> 2013-04-03 10:36:10.714542 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos latest full 393798 -18963> 2013-04-03 10:36:10.714561 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394004 update_from_paxos applying incremental log 394004 2013-04-03 10:36:09.648975 mon.0 10.252.24.25:6789/0 60 : [INF] pgmap v28998: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18962> 2013-04-03 10:36:10.714602 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 check_subs -18961> 2013-04-03 10:36:10.714632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 create_pending v 394005 -18960> 2013-04-03 10:36:10.714653 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394004 _updated_log for mon.0 10.252.24.25:6789/0 -18959> 2013-04-03 10:36:10.714665 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 60) v1 -- ?+0 0x2224380 con 0x1bac160 -18958> 2013-04-03 10:36:10.714685 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos -18957> 2013-04-03 10:36:10.714700 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394004 -18956> 2013-04-03 10:36:10.714708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18955> 2013-04-03 10:36:10.714719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 preprocess_log log(1 entries) v1 from mon.0 -18954> 2013-04-03 10:36:10.714734 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 nothing new -18953> 2013-04-03 10:36:10.714782 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos -18952> 2013-04-03 10:36:10.714797 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394004 -18951> 2013-04-03 10:36:10.714814 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18950> 2013-04-03 10:36:10.714835 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 60) v1 ==== 0+0+0 (0 0 0) 0x2224380 con 0x1bac160 -18949> 2013-04-03 10:36:10.714842 7fcdffcdb700 10 handle_log_ack log(last 60) v1 -18948> 2013-04-03 10:36:10.714845 7fcdffcdb700 10 logged 2013-04-03 10:36:09.648975 mon.0 10.252.24.25:6789/0 60 : [INF] pgmap v28998: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18947> 2013-04-03 10:36:10.726595 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113159 ==== paxos(lease_ack lc 427282 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (872956031 0 0) 0x26dc500 con 0x208a840 -18946> 2013-04-03 10:36:10.726629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos -18945> 2013-04-03 10:36:10.726645 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394004 -18944> 2013-04-03 10:36:10.726660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18943> 2013-04-03 10:36:11.579635 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 38 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140326 v187) v2 ==== 289+0+0 (3852265232 0 0) 0x273adc0 con 0x2118420 -18942> 2013-04-03 10:36:11.579708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140326 v187) v2 from mds.? 10.252.58.31:6800/10908 -18941> 2013-04-03 10:36:11.579721 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140326 v187) v2 -- ?+0 0x29c5080 con 0x2118420 -18940> 2013-04-03 10:36:12.925578 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113160 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 41 v187) v2) to leader v1 ==== 560+0+0 (3845457846 0 0) 0x29c4dc0 con 0x208a840 -18939> 2013-04-03 10:36:12.925598 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -18938> 2013-04-03 10:36:12.925608 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c4b00 from 10.252.24.88:6789/0 -18937> 2013-04-03 10:36:12.925660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 41 v187) v2 from mds.0 10.252.24.25:6802/3144 -18936> 2013-04-03 10:36:12.925671 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 41 v187) v2 tid 17467) v2 -- ?+0 0x24f8900 con 0x208a840 -18935> 2013-04-03 10:36:13.330661 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28999 check_down_pgs -18934> 2013-04-03 10:36:13.330687 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28999 v28999: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s -18933> 2013-04-03 10:36:13.330713 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -18932> 2013-04-03 10:36:13.330737 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -18931> 2013-04-03 10:36:13.330774 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -18930> 2013-04-03 10:36:13.330804 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos -18929> 2013-04-03 10:36:13.330823 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394004 -18928> 2013-04-03 10:36:13.330832 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394004 log -18927> 2013-04-03 10:36:13.330848 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18926> 2013-04-03 10:36:13.330866 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -18925> 2013-04-03 10:36:13.379464 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 38 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21697 v187) v2 ==== 289+0+0 (1100351458 0 0) 0x29952c0 con 0x200d420 -18924> 2013-04-03 10:36:13.379510 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21697 v187) v2 from mds.? 10.252.24.88:6800/17213 -18923> 2013-04-03 10:36:13.379521 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21697 v187) v2 -- ?+0 0x29c4dc0 con 0x200d420 -18922> 2013-04-03 10:36:13.487532 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113161 ==== forward(pg_stats(1 pgs tid 8 v 602) v1) to leader v1 ==== 757+0+0 (1875228213 0 0) 0x273adc0 con 0x208a840 -18921> 2013-04-03 10:36:13.487546 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 -18920> 2013-04-03 10:36:13.487554 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -18919> 2013-04-03 10:36:13.487583 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 preprocess_query pg_stats(1 pgs tid 8 v 602) v1 from osd.2 10.252.24.30:6800/1508 -18918> 2013-04-03 10:36:13.487604 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 prepare_update pg_stats(1 pgs tid 8 v 602) v1 from osd.2 10.252.24.30:6800/1508 -18917> 2013-04-03 10:36:13.487609 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 prepare_pg_stats pg_stats(1 pgs tid 8 v 602) v1 from osd.2 -18916> 2013-04-03 10:36:13.487615 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v28999 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -18915> 2013-04-03 10:36:13.537793 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v28999 encode_pending v 29000 -18914> 2013-04-03 10:36:13.537827 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427282) queue_proposal bl 654 bytes; ctx = 0x1b91d30 -18913> 2013-04-03 10:36:13.537836 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427282) propose_queued 427283 654 bytes -18912> 2013-04-03 10:36:13.538769 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427282 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0c80 -18911> 2013-04-03 10:36:13.582605 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113162 ==== paxos(accept lc 427282 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1628737592 0 0) 0x29e0c80 con 0x208a840 -18910> 2013-04-03 10:36:13.582903 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427283 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc500 -18909> 2013-04-03 10:36:13.582935 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427283 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29b4780 -18908> 2013-04-03 10:36:13.583007 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v28999 update_from_paxos applying incremental 29000 -18907> 2013-04-03 10:36:13.583023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 v29000: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3394B/s wr, 0op/s -18906> 2013-04-03 10:36:13.583039 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 send_pg_creates to 0 pgs -18905> 2013-04-03 10:36:13.583042 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 update_logger -18904> 2013-04-03 10:36:13.583051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 create_pending v 29001 -18903> 2013-04-03 10:36:13.583059 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29000 _updated_stats for osd.2 10.252.24.30:6800/1508 -18902> 2013-04-03 10:36:13.583066 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 8) v1 tid 17468) v2 -- ?+0 0x20adb40 con 0x208a840 -18901> 2013-04-03 10:36:13.583079 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 check_osd_map already seen 602 >= 602 -18900> 2013-04-03 10:36:13.583082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 update_logger -18899> 2013-04-03 10:36:13.583092 7fcdffcdb700 0 log [INF] : pgmap v29000: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3394B/s wr, 0op/s -18898> 2013-04-03 10:36:13.583093 7fcdffcdb700 10 send_log to self -18897> 2013-04-03 10:36:13.583094 7fcdffcdb700 10 log_queue is 2 last_log 62 sent 60 num 2 unsent 2 sending 2 -18896> 2013-04-03 10:36:13.583096 7fcdffcdb700 10 will send 2013-04-03 10:36:10.676807 mon.0 10.252.24.25:6789/0 61 : [INF] pgmap v28999: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s -18895> 2013-04-03 10:36:13.583102 7fcdffcdb700 10 will send 2013-04-03 10:36:13.583093 mon.0 10.252.24.25:6789/0 62 : [INF] pgmap v29000: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3394B/s wr, 0op/s -18894> 2013-04-03 10:36:13.583107 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2874800 -18893> 2013-04-03 10:36:13.583153 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos -18892> 2013-04-03 10:36:13.583171 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394004 -18891> 2013-04-03 10:36:13.583187 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18890> 2013-04-03 10:36:13.583208 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2874800 con 0x1bac160 -18889> 2013-04-03 10:36:13.583229 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos -18888> 2013-04-03 10:36:13.583244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394004 -18887> 2013-04-03 10:36:13.583253 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -18886> 2013-04-03 10:36:13.583265 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 preprocess_log log(2 entries) v1 from mon.0 -18885> 2013-04-03 10:36:13.583292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -18884> 2013-04-03 10:36:13.583304 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 prepare_log log(2 entries) v1 from mon.0 -18883> 2013-04-03 10:36:13.583315 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 logging 2013-04-03 10:36:10.676807 mon.0 10.252.24.25:6789/0 61 : [INF] pgmap v28999: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s -18882> 2013-04-03 10:36:13.583338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 logging 2013-04-03 10:36:13.583093 mon.0 10.252.24.25:6789/0 62 : [INF] pgmap v29000: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3394B/s wr, 0op/s -18881> 2013-04-03 10:36:13.594500 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113163 ==== paxos(lease_ack lc 427283 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (33614596 0 0) 0x29b4780 con 0x208a840 -18880> 2013-04-03 10:36:13.594536 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos -18879> 2013-04-03 10:36:13.594552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394004 update_from_paxos version 394004 summary v 394004 -18878> 2013-04-03 10:36:13.594567 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18877> 2013-04-03 10:36:14.582803 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394004 encode_pending v394005 -18876> 2013-04-03 10:36:14.582831 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427283) queue_proposal bl 725 bytes; ctx = 0x1b91ca0 -18875> 2013-04-03 10:36:14.582843 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427283) propose_queued 427284 725 bytes -18874> 2013-04-03 10:36:14.583893 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427283 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0a00 -18873> 2013-04-03 10:36:14.610396 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113164 ==== paxos(accept lc 427283 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (684170446 0 0) 0x29e0a00 con 0x208a840 -18872> 2013-04-03 10:36:14.610702 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427284 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4780 -18871> 2013-04-03 10:36:14.610732 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427284 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29e0c80 -18870> 2013-04-03 10:36:14.610772 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos -18869> 2013-04-03 10:36:14.610799 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394004 -18868> 2013-04-03 10:36:14.610815 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos latest full 393798 -18867> 2013-04-03 10:36:14.610834 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394005 update_from_paxos applying incremental log 394005 2013-04-03 10:36:10.676807 mon.0 10.252.24.25:6789/0 61 : [INF] pgmap v28999: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s -18866> 2013-04-03 10:36:14.610861 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394005 update_from_paxos applying incremental log 394005 2013-04-03 10:36:13.583093 mon.0 10.252.24.25:6789/0 62 : [INF] pgmap v29000: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3394B/s wr, 0op/s -18865> 2013-04-03 10:36:14.610898 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 check_subs -18864> 2013-04-03 10:36:14.610930 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 create_pending v 394006 -18863> 2013-04-03 10:36:14.610952 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394005 _updated_log for mon.0 10.252.24.25:6789/0 -18862> 2013-04-03 10:36:14.610964 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 62) v1 -- ?+0 0x2224c40 con 0x1bac160 -18861> 2013-04-03 10:36:14.611014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos -18860> 2013-04-03 10:36:14.611030 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 -18859> 2013-04-03 10:36:14.611050 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18858> 2013-04-03 10:36:14.611073 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 62) v1 ==== 0+0+0 (0 0 0) 0x2224c40 con 0x1bac160 -18857> 2013-04-03 10:36:14.611081 7fcdffcdb700 10 handle_log_ack log(last 62) v1 -18856> 2013-04-03 10:36:14.611083 7fcdffcdb700 10 logged 2013-04-03 10:36:10.676807 mon.0 10.252.24.25:6789/0 61 : [INF] pgmap v28999: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 719B/s wr, 0op/s -18855> 2013-04-03 10:36:14.611089 7fcdffcdb700 10 logged 2013-04-03 10:36:13.583093 mon.0 10.252.24.25:6789/0 62 : [INF] pgmap v29000: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3394B/s wr, 0op/s -18854> 2013-04-03 10:36:14.622360 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113165 ==== paxos(lease_ack lc 427284 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (2729556550 0 0) 0x29e0c80 con 0x208a840 -18853> 2013-04-03 10:36:14.622395 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos -18852> 2013-04-03 10:36:14.622411 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 -18851> 2013-04-03 10:36:14.622425 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18850> 2013-04-03 10:36:15.579759 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 39 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140327 v187) v2 ==== 289+0+0 (3337771117 0 0) 0x273b340 con 0x2118420 -18849> 2013-04-03 10:36:15.579830 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140327 v187) v2 from mds.? 10.252.58.31:6800/10908 -18848> 2013-04-03 10:36:15.579842 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140327 v187) v2 -- ?+0 0x273adc0 con 0x2118420 -18847> 2013-04-03 10:36:16.925725 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113166 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 42 v187) v2) to leader v1 ==== 560+0+0 (2269544120 0 0) 0x29c4840 con 0x208a840 -18846> 2013-04-03 10:36:16.925747 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -18845> 2013-04-03 10:36:16.925758 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c4580 from 10.252.24.88:6789/0 -18844> 2013-04-03 10:36:16.925809 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 42 v187) v2 from mds.0 10.252.24.25:6802/3144 -18843> 2013-04-03 10:36:16.925820 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 42 v187) v2 tid 17469) v2 -- ?+0 0x20fc240 con 0x208a840 -18842> 2013-04-03 10:36:17.379601 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 39 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21698 v187) v2 ==== 289+0+0 (606455397 0 0) 0x2995000 con 0x200d420 -18841> 2013-04-03 10:36:17.379679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21698 v187) v2 from mds.? 10.252.24.88:6800/17213 -18840> 2013-04-03 10:36:17.379694 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21698 v187) v2 -- ?+0 0x29c4840 con 0x200d420 -18839> 2013-04-03 10:36:17.414618 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 40 ==== mon_subscribe({mdsmap=188+,monmap=6+}) v2 ==== 42+0+0 (435898880 0 0) 0x2294a80 con 0x200d420 -18838> 2013-04-03 10:36:17.414630 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 -18837> 2013-04-03 10:36:17.414637 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -18836> 2013-04-03 10:36:17.414639 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/17213 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2224c40 -18835> 2013-04-03 10:36:17.610803 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427284 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29e0780 -18834> 2013-04-03 10:36:17.611583 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113167 ==== paxos(lease_ack lc 427284 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (2426360190 0 0) 0x29e0780 con 0x208a840 -18833> 2013-04-03 10:36:17.611831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos -18832> 2013-04-03 10:36:17.611854 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 -18831> 2013-04-03 10:36:17.611871 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18830> 2013-04-03 10:36:18.331011 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29000 check_down_pgs -18829> 2013-04-03 10:36:18.331042 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29000 v29000: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3394B/s wr, 0op/s -18828> 2013-04-03 10:36:18.331069 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -18827> 2013-04-03 10:36:18.331093 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -18826> 2013-04-03 10:36:18.331129 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -18825> 2013-04-03 10:36:18.331161 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos -18824> 2013-04-03 10:36:18.331179 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 -18823> 2013-04-03 10:36:18.331188 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394005 log -18822> 2013-04-03 10:36:18.331205 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18821> 2013-04-03 10:36:18.331223 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -18820> 2013-04-03 10:36:19.579889 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 40 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140328 v187) v2 ==== 289+0+0 (811788551 0 0) 0x273b600 con 0x2118420 -18819> 2013-04-03 10:36:19.579962 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140328 v187) v2 from mds.? 10.252.58.31:6800/10908 -18818> 2013-04-03 10:36:19.579975 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140328 v187) v2 -- ?+0 0x2995000 con 0x2118420 -18817> 2013-04-03 10:36:20.610878 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427284 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29e0500 -18816> 2013-04-03 10:36:20.611686 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113168 ==== paxos(lease_ack lc 427284 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (1683921801 0 0) 0x29e0500 con 0x208a840 -18815> 2013-04-03 10:36:20.611880 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos -18814> 2013-04-03 10:36:20.611904 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 -18813> 2013-04-03 10:36:20.611922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18812> 2013-04-03 10:36:20.925804 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113169 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 43 v187) v2) to leader v1 ==== 560+0+0 (371667679 0 0) 0x273b340 con 0x208a840 -18811> 2013-04-03 10:36:20.925820 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -18810> 2013-04-03 10:36:20.925828 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c42c0 from 10.252.24.88:6789/0 -18809> 2013-04-03 10:36:20.925864 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 43 v187) v2 from mds.0 10.252.24.25:6802/3144 -18808> 2013-04-03 10:36:20.925875 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 43 v187) v2 tid 17470) v2 -- ?+0 0x20adb40 con 0x208a840 -18807> 2013-04-03 10:36:21.379770 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 41 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21699 v187) v2 ==== 289+0+0 (122386712 0 0) 0x2996b80 con 0x200d420 -18806> 2013-04-03 10:36:21.379839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21699 v187) v2 from mds.? 10.252.24.88:6800/17213 -18805> 2013-04-03 10:36:21.379852 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21699 v187) v2 -- ?+0 0x273b340 con 0x200d420 -18804> 2013-04-03 10:36:23.331365 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29000 v29000: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3394B/s wr, 0op/s -18803> 2013-04-03 10:36:23.331406 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -18802> 2013-04-03 10:36:23.331428 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -18801> 2013-04-03 10:36:23.331469 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -18800> 2013-04-03 10:36:23.331499 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos -18799> 2013-04-03 10:36:23.331517 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 -18798> 2013-04-03 10:36:23.331526 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394005 log -18797> 2013-04-03 10:36:23.331543 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18796> 2013-04-03 10:36:23.331565 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -18795> 2013-04-03 10:36:23.580011 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 41 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140329 v187) v2 ==== 289+0+0 (319575674 0 0) 0x29c5080 con 0x2118420 -18794> 2013-04-03 10:36:23.580076 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140329 v187) v2 from mds.? 10.252.58.31:6800/10908 -18793> 2013-04-03 10:36:23.580088 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140329 v187) v2 -- ?+0 0x2996b80 con 0x2118420 -18792> 2013-04-03 10:36:23.610946 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427284 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29e0280 -18791> 2013-04-03 10:36:23.611814 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113170 ==== paxos(lease_ack lc 427284 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (135054458 0 0) 0x29e0280 con 0x208a840 -18790> 2013-04-03 10:36:23.611860 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos -18789> 2013-04-03 10:36:23.611879 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 -18788> 2013-04-03 10:36:23.611895 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18787> 2013-04-03 10:36:24.567572 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113171 ==== forward(pg_stats(1 pgs tid 16 v 602) v1) to leader v1 ==== 757+0+0 (28138126 0 0) 0x273b600 con 0x208a840 -18786> 2013-04-03 10:36:24.567592 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -18785> 2013-04-03 10:36:24.567601 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -18784> 2013-04-03 10:36:24.567652 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 preprocess_query pg_stats(1 pgs tid 16 v 602) v1 from osd.1 10.252.24.25:6805/3572 -18783> 2013-04-03 10:36:24.567675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 prepare_update pg_stats(1 pgs tid 16 v 602) v1 from osd.1 10.252.24.25:6805/3572 -18782> 2013-04-03 10:36:24.567681 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 prepare_pg_stats pg_stats(1 pgs tid 16 v 602) v1 from osd.1 -18781> 2013-04-03 10:36:24.567687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29000 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -18780> 2013-04-03 10:36:24.617874 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29000 encode_pending v 29001 -18779> 2013-04-03 10:36:24.617909 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427284) queue_proposal bl 654 bytes; ctx = 0x1b91de0 -18778> 2013-04-03 10:36:24.617918 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427284) propose_queued 427285 654 bytes -18777> 2013-04-03 10:36:24.618837 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427284 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0000 -18776> 2013-04-03 10:36:24.646421 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113172 ==== paxos(accept lc 427284 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3242036766 0 0) 0x29e0000 con 0x208a840 -18775> 2013-04-03 10:36:24.657114 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427285 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0280 -18774> 2013-04-03 10:36:24.657147 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427285 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29e0500 -18773> 2013-04-03 10:36:24.657210 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29000 update_from_paxos applying incremental 29001 -18772> 2013-04-03 10:36:24.657226 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 v29001: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -18771> 2013-04-03 10:36:24.657245 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 send_pg_creates to 0 pgs -18770> 2013-04-03 10:36:24.657248 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 update_logger -18769> 2013-04-03 10:36:24.657257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 create_pending v 29002 -18768> 2013-04-03 10:36:24.657266 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29001 _updated_stats for osd.1 10.252.24.25:6805/3572 -18767> 2013-04-03 10:36:24.657273 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 16) v1 tid 17471) v2 -- ?+0 0x20fc240 con 0x208a840 -18766> 2013-04-03 10:36:24.657282 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 check_osd_map already seen 602 >= 602 -18765> 2013-04-03 10:36:24.657285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 update_logger -18764> 2013-04-03 10:36:24.657295 7fcdffcdb700 0 log [INF] : pgmap v29001: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -18763> 2013-04-03 10:36:24.657296 7fcdffcdb700 10 send_log to self -18762> 2013-04-03 10:36:24.657297 7fcdffcdb700 10 log_queue is 1 last_log 63 sent 62 num 1 unsent 1 sending 1 -18761> 2013-04-03 10:36:24.657299 7fcdffcdb700 10 will send 2013-04-03 10:36:24.657296 mon.0 10.252.24.25:6789/0 63 : [INF] pgmap v29001: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -18760> 2013-04-03 10:36:24.657305 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2874800 -18759> 2013-04-03 10:36:24.657362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos -18758> 2013-04-03 10:36:24.657382 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 -18757> 2013-04-03 10:36:24.657401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18756> 2013-04-03 10:36:24.657423 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2874800 con 0x1bac160 -18755> 2013-04-03 10:36:24.657444 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos -18754> 2013-04-03 10:36:24.657459 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 -18753> 2013-04-03 10:36:24.657468 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18752> 2013-04-03 10:36:24.657478 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 preprocess_log log(1 entries) v1 from mon.0 -18751> 2013-04-03 10:36:24.657500 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18750> 2013-04-03 10:36:24.657512 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 prepare_log log(1 entries) v1 from mon.0 -18749> 2013-04-03 10:36:24.657521 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 logging 2013-04-03 10:36:24.657296 mon.0 10.252.24.25:6789/0 63 : [INF] pgmap v29001: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -18748> 2013-04-03 10:36:24.667998 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113173 ==== paxos(lease_ack lc 427285 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (9063444 0 0) 0x29e0500 con 0x208a840 -18747> 2013-04-03 10:36:24.668033 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos -18746> 2013-04-03 10:36:24.668049 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394005 update_from_paxos version 394005 summary v 394005 -18745> 2013-04-03 10:36:24.668063 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18744> 2013-04-03 10:36:24.925922 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113174 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 44 v187) v2) to leader v1 ==== 560+0+0 (614912712 0 0) 0x29c4000 con 0x208a840 -18743> 2013-04-03 10:36:24.925935 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -18742> 2013-04-03 10:36:24.925943 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29f5b80 from 10.252.24.88:6789/0 -18741> 2013-04-03 10:36:24.925974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 44 v187) v2 from mds.0 10.252.24.25:6802/3144 -18740> 2013-04-03 10:36:24.925984 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 44 v187) v2 tid 17472) v2 -- ?+0 0x20adb40 con 0x208a840 -18739> 2013-04-03 10:36:25.379934 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 42 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21700 v187) v2 ==== 289+0+0 (4014097771 0 0) 0x273a580 con 0x200d420 -18738> 2013-04-03 10:36:25.380010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21700 v187) v2 from mds.? 10.252.24.88:6800/17213 -18737> 2013-04-03 10:36:25.380024 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21700 v187) v2 -- ?+0 0x29c4000 con 0x200d420 -18736> 2013-04-03 10:36:25.479031 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 42 ==== mon_subscribe({mdsmap=188+,monmap=6+}) v2 ==== 42+0+0 (435898880 0 0) 0x28a5c40 con 0x2118420 -18735> 2013-04-03 10:36:25.479045 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 -18734> 2013-04-03 10:36:25.479053 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -18733> 2013-04-03 10:36:25.479057 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.58.31:6800/10908 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2224380 -18732> 2013-04-03 10:36:25.646612 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394005 encode_pending v394006 -18731> 2013-04-03 10:36:25.646643 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427285) queue_proposal bl 443 bytes; ctx = 0x1b91cb0 -18730> 2013-04-03 10:36:25.646650 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427285) propose_queued 427286 443 bytes -18729> 2013-04-03 10:36:25.647739 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427285 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fdc80 -18728> 2013-04-03 10:36:25.766008 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113175 ==== paxos(accept lc 427285 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4225067362 0 0) 0x29fdc80 con 0x208a840 -18727> 2013-04-03 10:36:25.766368 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427286 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0500 -18726> 2013-04-03 10:36:25.766390 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427286 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29e0000 -18725> 2013-04-03 10:36:25.766433 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos -18724> 2013-04-03 10:36:25.766468 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos version 394006 summary v 394005 -18723> 2013-04-03 10:36:25.766482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos latest full 393798 -18722> 2013-04-03 10:36:25.766500 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394006 update_from_paxos applying incremental log 394006 2013-04-03 10:36:24.657296 mon.0 10.252.24.25:6789/0 63 : [INF] pgmap v29001: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -18721> 2013-04-03 10:36:25.766551 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 check_subs -18720> 2013-04-03 10:36:25.766582 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 create_pending v 394007 -18719> 2013-04-03 10:36:25.766603 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394006 _updated_log for mon.0 10.252.24.25:6789/0 -18718> 2013-04-03 10:36:25.766616 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 63) v1 -- ?+0 0x28a5c40 con 0x1bac160 -18717> 2013-04-03 10:36:25.766666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos -18716> 2013-04-03 10:36:25.766681 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos version 394006 summary v 394006 -18715> 2013-04-03 10:36:25.766702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18714> 2013-04-03 10:36:25.766723 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 63) v1 ==== 0+0+0 (0 0 0) 0x28a5c40 con 0x1bac160 -18713> 2013-04-03 10:36:25.766731 7fcdffcdb700 10 handle_log_ack log(last 63) v1 -18712> 2013-04-03 10:36:25.766733 7fcdffcdb700 10 logged 2013-04-03 10:36:24.657296 mon.0 10.252.24.25:6789/0 63 : [INF] pgmap v29001: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -18711> 2013-04-03 10:36:25.779157 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113176 ==== paxos(lease_ack lc 427286 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (1306560777 0 0) 0x29e0000 con 0x208a840 -18710> 2013-04-03 10:36:25.779194 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos -18709> 2013-04-03 10:36:25.779211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos version 394006 summary v 394006 -18708> 2013-04-03 10:36:25.779225 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18707> 2013-04-03 10:36:27.580128 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 43 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140330 v187) v2 ==== 289+0+0 (1992267773 0 0) 0x29c4840 con 0x2118420 -18706> 2013-04-03 10:36:27.580198 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140330 v187) v2 from mds.? 10.252.58.31:6800/10908 -18705> 2013-04-03 10:36:27.580211 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140330 v187) v2 -- ?+0 0x273a580 con 0x2118420 -18704> 2013-04-03 10:36:28.331711 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29001 check_down_pgs -18703> 2013-04-03 10:36:28.331735 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29001 v29001: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -18702> 2013-04-03 10:36:28.331761 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -18701> 2013-04-03 10:36:28.331785 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -18700> 2013-04-03 10:36:28.331821 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -18699> 2013-04-03 10:36:28.331851 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos -18698> 2013-04-03 10:36:28.331869 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos version 394006 summary v 394006 -18697> 2013-04-03 10:36:28.331879 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394006 log -18696> 2013-04-03 10:36:28.331895 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18695> 2013-04-03 10:36:28.331917 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -18694> 2013-04-03 10:36:28.489567 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113177 ==== forward(pg_stats(1 pgs tid 9 v 602) v1) to leader v1 ==== 757+0+0 (2826663982 0 0) 0x273b600 con 0x208a840 -18693> 2013-04-03 10:36:28.489587 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 -18692> 2013-04-03 10:36:28.489596 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -18691> 2013-04-03 10:36:28.489631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 preprocess_query pg_stats(1 pgs tid 9 v 602) v1 from osd.2 10.252.24.30:6800/1508 -18690> 2013-04-03 10:36:28.489653 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 prepare_update pg_stats(1 pgs tid 9 v 602) v1 from osd.2 10.252.24.30:6800/1508 -18689> 2013-04-03 10:36:28.489658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 prepare_pg_stats pg_stats(1 pgs tid 9 v 602) v1 from osd.2 -18688> 2013-04-03 10:36:28.489664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29001 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -18687> 2013-04-03 10:36:28.539849 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29001 encode_pending v 29002 -18686> 2013-04-03 10:36:28.539886 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427286) queue_proposal bl 654 bytes; ctx = 0x1b91d00 -18685> 2013-04-03 10:36:28.539895 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427286) propose_queued 427287 654 bytes -18684> 2013-04-03 10:36:28.541121 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427286 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fda00 -18683> 2013-04-03 10:36:28.567306 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113178 ==== paxos(accept lc 427286 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3144326113 0 0) 0x29fda00 con 0x208a840 -18682> 2013-04-03 10:36:28.573159 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427287 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0000 -18681> 2013-04-03 10:36:28.573186 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427287 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29fdc80 -18680> 2013-04-03 10:36:28.573257 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29001 update_from_paxos applying incremental 29002 -18679> 2013-04-03 10:36:28.573272 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 v29002: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18678> 2013-04-03 10:36:28.573290 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 send_pg_creates to 0 pgs -18677> 2013-04-03 10:36:28.573294 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 update_logger -18676> 2013-04-03 10:36:28.573304 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 create_pending v 29003 -18675> 2013-04-03 10:36:28.573313 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29002 _updated_stats for osd.2 10.252.24.30:6800/1508 -18674> 2013-04-03 10:36:28.573319 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 9) v1 tid 17473) v2 -- ?+0 0x20fc240 con 0x208a840 -18673> 2013-04-03 10:36:28.573328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 check_osd_map already seen 602 >= 602 -18672> 2013-04-03 10:36:28.573331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 update_logger -18671> 2013-04-03 10:36:28.573341 7fcdffcdb700 0 log [INF] : pgmap v29002: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18670> 2013-04-03 10:36:28.573343 7fcdffcdb700 10 send_log to self -18669> 2013-04-03 10:36:28.573343 7fcdffcdb700 10 log_queue is 1 last_log 64 sent 63 num 1 unsent 1 sending 1 -18668> 2013-04-03 10:36:28.573345 7fcdffcdb700 10 will send 2013-04-03 10:36:28.573342 mon.0 10.252.24.25:6789/0 64 : [INF] pgmap v29002: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18667> 2013-04-03 10:36:28.573351 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2874800 -18666> 2013-04-03 10:36:28.573394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos -18665> 2013-04-03 10:36:28.573411 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos version 394006 summary v 394006 -18664> 2013-04-03 10:36:28.573428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18663> 2013-04-03 10:36:28.573450 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2874800 con 0x1bac160 -18662> 2013-04-03 10:36:28.573472 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos -18661> 2013-04-03 10:36:28.573486 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos version 394006 summary v 394006 -18660> 2013-04-03 10:36:28.573494 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18659> 2013-04-03 10:36:28.573504 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 preprocess_log log(1 entries) v1 from mon.0 -18658> 2013-04-03 10:36:28.573524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18657> 2013-04-03 10:36:28.573535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 prepare_log log(1 entries) v1 from mon.0 -18656> 2013-04-03 10:36:28.573546 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 logging 2013-04-03 10:36:28.573342 mon.0 10.252.24.25:6789/0 64 : [INF] pgmap v29002: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18655> 2013-04-03 10:36:28.588880 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113179 ==== paxos(lease_ack lc 427287 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (4021956847 0 0) 0x29fdc80 con 0x208a840 -18654> 2013-04-03 10:36:28.588917 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos -18653> 2013-04-03 10:36:28.588933 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394006 update_from_paxos version 394006 summary v 394006 -18652> 2013-04-03 10:36:28.588947 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18651> 2013-04-03 10:36:28.926101 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113180 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 45 v187) v2) to leader v1 ==== 560+0+0 (4191410077 0 0) 0x29f58c0 con 0x208a840 -18650> 2013-04-03 10:36:28.926115 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -18649> 2013-04-03 10:36:28.926123 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29f5600 from 10.252.24.88:6789/0 -18648> 2013-04-03 10:36:28.926154 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 45 v187) v2 from mds.0 10.252.24.25:6802/3144 -18647> 2013-04-03 10:36:28.926436 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 45 v187) v2 tid 17474) v2 -- ?+0 0x20adb40 con 0x208a840 -18646> 2013-04-03 10:36:29.380000 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 43 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21701 v187) v2 ==== 289+0+0 (3425499670 0 0) 0x273a2c0 con 0x200d420 -18645> 2013-04-03 10:36:29.380077 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21701 v187) v2 from mds.? 10.252.24.88:6800/17213 -18644> 2013-04-03 10:36:29.380094 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21701 v187) v2 -- ?+0 0x29f58c0 con 0x200d420 -18643> 2013-04-03 10:36:29.567486 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394006 encode_pending v394007 -18642> 2013-04-03 10:36:29.567516 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427287) queue_proposal bl 443 bytes; ctx = 0x1b91ce0 -18641> 2013-04-03 10:36:29.567524 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427287) propose_queued 427288 443 bytes -18640> 2013-04-03 10:36:29.568587 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427287 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd780 -18639> 2013-04-03 10:36:29.596422 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113181 ==== paxos(accept lc 427287 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3744715866 0 0) 0x29fd780 con 0x208a840 -18638> 2013-04-03 10:36:29.596722 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427288 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fdc80 -18637> 2013-04-03 10:36:29.596757 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427288 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29fda00 -18636> 2013-04-03 10:36:29.596789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos -18635> 2013-04-03 10:36:29.596817 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394006 -18634> 2013-04-03 10:36:29.596831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos latest full 393798 -18633> 2013-04-03 10:36:29.596849 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394007 update_from_paxos applying incremental log 394007 2013-04-03 10:36:28.573342 mon.0 10.252.24.25:6789/0 64 : [INF] pgmap v29002: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18632> 2013-04-03 10:36:29.596894 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 check_subs -18631> 2013-04-03 10:36:29.596924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 create_pending v 394008 -18630> 2013-04-03 10:36:29.596946 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394007 _updated_log for mon.0 10.252.24.25:6789/0 -18629> 2013-04-03 10:36:29.596959 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 64) v1 -- ?+0 0x2294000 con 0x1bac160 -18628> 2013-04-03 10:36:29.597008 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos -18627> 2013-04-03 10:36:29.597023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 -18626> 2013-04-03 10:36:29.597041 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18625> 2013-04-03 10:36:29.597062 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 64) v1 ==== 0+0+0 (0 0 0) 0x2294000 con 0x1bac160 -18624> 2013-04-03 10:36:29.597070 7fcdffcdb700 10 handle_log_ack log(last 64) v1 -18623> 2013-04-03 10:36:29.597073 7fcdffcdb700 10 logged 2013-04-03 10:36:28.573342 mon.0 10.252.24.25:6789/0 64 : [INF] pgmap v29002: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18622> 2013-04-03 10:36:29.621638 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113182 ==== paxos(lease_ack lc 427288 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (3508945771 0 0) 0x29fda00 con 0x208a840 -18621> 2013-04-03 10:36:29.621675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos -18620> 2013-04-03 10:36:29.621692 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 -18619> 2013-04-03 10:36:29.621706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18618> 2013-04-03 10:36:31.580244 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 44 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140331 v187) v2 ==== 289+0+0 (1439770752 0 0) 0x29c4dc0 con 0x2118420 -18617> 2013-04-03 10:36:31.580319 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140331 v187) v2 from mds.? 10.252.58.31:6800/10908 -18616> 2013-04-03 10:36:31.580331 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140331 v187) v2 -- ?+0 0x273a2c0 con 0x2118420 -18615> 2013-04-03 10:36:32.596834 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427288 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29fd500 -18614> 2013-04-03 10:36:32.597637 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113183 ==== paxos(lease_ack lc 427288 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (2614503852 0 0) 0x29fd500 con 0x208a840 -18613> 2013-04-03 10:36:32.597702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos -18612> 2013-04-03 10:36:32.597722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 -18611> 2013-04-03 10:36:32.597738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18610> 2013-04-03 10:36:32.926205 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113184 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 46 v187) v2) to leader v1 ==== 560+0+0 (2048993276 0 0) 0x273b600 con 0x208a840 -18609> 2013-04-03 10:36:32.926220 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -18608> 2013-04-03 10:36:32.926229 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29f5340 from 10.252.24.88:6789/0 -18607> 2013-04-03 10:36:32.926263 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 46 v187) v2 from mds.0 10.252.24.25:6802/3144 -18606> 2013-04-03 10:36:32.926274 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 46 v187) v2 tid 17475) v2 -- ?+0 0x20fc240 con 0x208a840 -18605> 2013-04-03 10:36:33.332024 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29002 check_down_pgs -18604> 2013-04-03 10:36:33.332052 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29002 v29002: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18603> 2013-04-03 10:36:33.332078 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -18602> 2013-04-03 10:36:33.332102 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -18601> 2013-04-03 10:36:33.332136 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -18600> 2013-04-03 10:36:33.332166 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos -18599> 2013-04-03 10:36:33.332185 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 -18598> 2013-04-03 10:36:33.332194 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394007 log -18597> 2013-04-03 10:36:33.332210 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18596> 2013-04-03 10:36:33.332231 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -18595> 2013-04-03 10:36:33.380107 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 44 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21702 v187) v2 ==== 289+0+0 (2845784977 0 0) 0x273b8c0 con 0x200d420 -18594> 2013-04-03 10:36:33.380152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21702 v187) v2 from mds.? 10.252.24.88:6800/17213 -18593> 2013-04-03 10:36:33.380163 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21702 v187) v2 -- ?+0 0x273b600 con 0x200d420 -18592> 2013-04-03 10:36:35.580372 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 45 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140332 v187) v2 ==== 289+0+0 (3185054963 0 0) 0x29c5b80 con 0x2118420 -18591> 2013-04-03 10:36:35.580445 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140332 v187) v2 from mds.? 10.252.58.31:6800/10908 -18590> 2013-04-03 10:36:35.580458 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140332 v187) v2 -- ?+0 0x273b8c0 con 0x2118420 -18589> 2013-04-03 10:36:35.596899 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427288 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29fd280 -18588> 2013-04-03 10:36:35.597670 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113185 ==== paxos(lease_ack lc 427288 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (1388543551 0 0) 0x29fd280 con 0x208a840 -18587> 2013-04-03 10:36:35.597714 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos -18586> 2013-04-03 10:36:35.597733 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 -18585> 2013-04-03 10:36:35.597749 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18584> 2013-04-03 10:36:36.926329 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113186 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 47 v187) v2) to leader v1 ==== 560+0+0 (3741137314 0 0) 0x29c4dc0 con 0x208a840 -18583> 2013-04-03 10:36:36.926349 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -18582> 2013-04-03 10:36:36.926358 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29f5080 from 10.252.24.88:6789/0 -18581> 2013-04-03 10:36:36.926409 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 47 v187) v2 from mds.0 10.252.24.25:6802/3144 -18580> 2013-04-03 10:36:36.926421 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 47 v187) v2 tid 17476) v2 -- ?+0 0x20adb40 con 0x208a840 -18579> 2013-04-03 10:36:37.380218 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 45 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21703 v187) v2 ==== 289+0+0 (2331084012 0 0) 0x273a000 con 0x200d420 -18578> 2013-04-03 10:36:37.380294 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21703 v187) v2 from mds.? 10.252.24.88:6800/17213 -18577> 2013-04-03 10:36:37.380307 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21703 v187) v2 -- ?+0 0x29c4dc0 con 0x200d420 -18576> 2013-04-03 10:36:38.332378 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29002 v29002: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18575> 2013-04-03 10:36:38.332417 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -18574> 2013-04-03 10:36:38.332439 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -18573> 2013-04-03 10:36:38.332479 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -18572> 2013-04-03 10:36:38.332510 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos -18571> 2013-04-03 10:36:38.332528 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 -18570> 2013-04-03 10:36:38.332537 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394007 log -18569> 2013-04-03 10:36:38.332554 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18568> 2013-04-03 10:36:38.332572 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -18567> 2013-04-03 10:36:38.596964 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427288 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29fd000 -18566> 2013-04-03 10:36:38.597750 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113187 ==== paxos(lease_ack lc 427288 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (2057309342 0 0) 0x29fd000 con 0x208a840 -18565> 2013-04-03 10:36:38.597806 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos -18564> 2013-04-03 10:36:38.597824 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 -18563> 2013-04-03 10:36:38.597839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18562> 2013-04-03 10:36:39.569228 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113188 ==== forward(pg_stats(1 pgs tid 17 v 602) v1) to leader v1 ==== 757+0+0 (451060357 0 0) 0x29c5b80 con 0x208a840 -18561> 2013-04-03 10:36:39.569250 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -18560> 2013-04-03 10:36:39.569259 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -18559> 2013-04-03 10:36:39.569309 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 preprocess_query pg_stats(1 pgs tid 17 v 602) v1 from osd.1 10.252.24.25:6805/3572 -18558> 2013-04-03 10:36:39.569333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 prepare_update pg_stats(1 pgs tid 17 v 602) v1 from osd.1 10.252.24.25:6805/3572 -18557> 2013-04-03 10:36:39.569338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 prepare_pg_stats pg_stats(1 pgs tid 17 v 602) v1 from osd.1 -18556> 2013-04-03 10:36:39.569345 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29002 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -18555> 2013-04-03 10:36:39.580460 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 46 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140333 v187) v2 ==== 289+0+0 (2662771598 0 0) 0x2995840 con 0x2118420 -18554> 2013-04-03 10:36:39.580516 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140333 v187) v2 from mds.? 10.252.58.31:6800/10908 -18553> 2013-04-03 10:36:39.580527 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140333 v187) v2 -- ?+0 0x29c5b80 con 0x2118420 -18552> 2013-04-03 10:36:39.619525 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29002 encode_pending v 29003 -18551> 2013-04-03 10:36:39.619561 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427288) queue_proposal bl 654 bytes; ctx = 0x1b91d40 -18550> 2013-04-03 10:36:39.619570 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427288) propose_queued 427289 654 bytes -18549> 2013-04-03 10:36:39.620727 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427288 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd500 -18548> 2013-04-03 10:36:39.663757 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113189 ==== paxos(accept lc 427288 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3654563045 0 0) 0x29fd500 con 0x208a840 -18547> 2013-04-03 10:36:39.664069 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427289 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd000 -18546> 2013-04-03 10:36:39.664100 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427289 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29fd280 -18545> 2013-04-03 10:36:39.664170 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29002 update_from_paxos applying incremental 29003 -18544> 2013-04-03 10:36:39.664185 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 v29003: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18543> 2013-04-03 10:36:39.664202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 send_pg_creates to 0 pgs -18542> 2013-04-03 10:36:39.664205 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 update_logger -18541> 2013-04-03 10:36:39.664214 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 create_pending v 29004 -18540> 2013-04-03 10:36:39.664222 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29003 _updated_stats for osd.1 10.252.24.25:6805/3572 -18539> 2013-04-03 10:36:39.664228 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 17) v1 tid 17477) v2 -- ?+0 0x20fc240 con 0x208a840 -18538> 2013-04-03 10:36:39.664239 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 check_osd_map already seen 602 >= 602 -18537> 2013-04-03 10:36:39.664241 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 update_logger -18536> 2013-04-03 10:36:39.664251 7fcdffcdb700 0 log [INF] : pgmap v29003: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18535> 2013-04-03 10:36:39.664253 7fcdffcdb700 10 send_log to self -18534> 2013-04-03 10:36:39.664254 7fcdffcdb700 10 log_queue is 1 last_log 65 sent 64 num 1 unsent 1 sending 1 -18533> 2013-04-03 10:36:39.664255 7fcdffcdb700 10 will send 2013-04-03 10:36:39.664253 mon.0 10.252.24.25:6789/0 65 : [INF] pgmap v29003: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18532> 2013-04-03 10:36:39.664262 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2874800 -18531> 2013-04-03 10:36:39.664306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos -18530> 2013-04-03 10:36:39.664325 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 -18529> 2013-04-03 10:36:39.664341 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18528> 2013-04-03 10:36:39.664363 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2874800 con 0x1bac160 -18527> 2013-04-03 10:36:39.664383 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos -18526> 2013-04-03 10:36:39.664397 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 -18525> 2013-04-03 10:36:39.664405 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18524> 2013-04-03 10:36:39.664415 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 preprocess_log log(1 entries) v1 from mon.0 -18523> 2013-04-03 10:36:39.664435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18522> 2013-04-03 10:36:39.664446 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 prepare_log log(1 entries) v1 from mon.0 -18521> 2013-04-03 10:36:39.664456 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 logging 2013-04-03 10:36:39.664253 mon.0 10.252.24.25:6789/0 65 : [INF] pgmap v29003: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18520> 2013-04-03 10:36:39.675759 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113190 ==== paxos(lease_ack lc 427289 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (462781092 0 0) 0x29fd280 con 0x208a840 -18519> 2013-04-03 10:36:39.675794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos -18518> 2013-04-03 10:36:39.675810 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394007 update_from_paxos version 394007 summary v 394007 -18517> 2013-04-03 10:36:39.675824 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18516> 2013-04-03 10:36:39.735834 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113191 ==== forward(pg_stats(1 pgs tid 30439 v 602) v1) to leader v1 ==== 757+0+0 (1381748069 0 0) 0x29f4dc0 con 0x208a840 -18515> 2013-04-03 10:36:39.735845 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -18514> 2013-04-03 10:36:39.735852 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 -18513> 2013-04-03 10:36:39.735877 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 preprocess_query pg_stats(1 pgs tid 30439 v 602) v1 from osd.3 10.252.24.88:6801/17538 -18512> 2013-04-03 10:36:39.735897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 prepare_update pg_stats(1 pgs tid 30439 v 602) v1 from osd.3 10.252.24.88:6801/17538 -18511> 2013-04-03 10:36:39.735903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 prepare_pg_stats pg_stats(1 pgs tid 30439 v 602) v1 from osd.3 -18510> 2013-04-03 10:36:39.735907 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29003 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -18509> 2013-04-03 10:36:40.663953 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29003 encode_pending v 29004 -18508> 2013-04-03 10:36:40.663985 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427289) queue_proposal bl 654 bytes; ctx = 0x1b91c80 -18507> 2013-04-03 10:36:40.663992 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427289) propose_queued 427290 654 bytes -18506> 2013-04-03 10:36:40.665143 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427289 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fda00 -18505> 2013-04-03 10:36:40.665240 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394007 encode_pending v394008 -18504> 2013-04-03 10:36:40.665267 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427258..427289) queue_proposal bl 443 bytes; ctx = 0x1b91dd0 -18503> 2013-04-03 10:36:40.665273 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427258..427289) propose_new_value not active; proposal queued -18502> 2013-04-03 10:36:40.708259 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113192 ==== paxos(accept lc 427289 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3233552662 0 0) 0x29fda00 con 0x208a840 -18501> 2013-04-03 10:36:40.708572 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427290 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd280 -18500> 2013-04-03 10:36:40.708605 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427290 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29fd500 -18499> 2013-04-03 10:36:40.708681 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29003 update_from_paxos applying incremental 29004 -18498> 2013-04-03 10:36:40.708699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 v29004: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 422B/s wr, 0op/s -18497> 2013-04-03 10:36:40.708719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 send_pg_creates to 0 pgs -18496> 2013-04-03 10:36:40.708722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 update_logger -18495> 2013-04-03 10:36:40.708731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 create_pending v 29005 -18494> 2013-04-03 10:36:40.708740 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29004 _updated_stats for osd.3 10.252.24.88:6801/17538 -18493> 2013-04-03 10:36:40.708747 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30439) v1 tid 17478) v2 -- ?+0 0x20adb40 con 0x208a840 -18492> 2013-04-03 10:36:40.708758 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 check_osd_map already seen 602 >= 602 -18491> 2013-04-03 10:36:40.708760 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 update_logger -18490> 2013-04-03 10:36:40.708772 7fcdffcdb700 0 log [INF] : pgmap v29004: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 422B/s wr, 0op/s -18489> 2013-04-03 10:36:40.708774 7fcdffcdb700 10 send_log to self -18488> 2013-04-03 10:36:40.708775 7fcdffcdb700 10 log_queue is 2 last_log 66 sent 65 num 2 unsent 1 sending 1 -18487> 2013-04-03 10:36:40.708776 7fcdffcdb700 10 will send 2013-04-03 10:36:39.664253 mon.0 10.252.24.25:6789/0 65 : [INF] pgmap v29003: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18486> 2013-04-03 10:36:40.708783 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287ee00 -18485> 2013-04-03 10:36:40.708812 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427290) propose_queued 427291 443 bytes -18484> 2013-04-03 10:36:40.709179 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427290 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2a2bc80 -18483> 2013-04-03 10:36:40.709197 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287ee00 con 0x1bac160 -18482> 2013-04-03 10:36:40.733451 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113193 ==== paxos(lease_ack lc 427290 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (2281544397 0 0) 0x2a2bc80 con 0x208a840 -18481> 2013-04-03 10:36:40.745373 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113194 ==== paxos(accept lc 427290 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3849006701 0 0) 0x29fd500 con 0x208a840 -18480> 2013-04-03 10:36:40.745644 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427291 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2a2bc80 -18479> 2013-04-03 10:36:40.745660 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427291 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29fda00 -18478> 2013-04-03 10:36:40.745693 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos -18477> 2013-04-03 10:36:40.745716 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394007 -18476> 2013-04-03 10:36:40.745730 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos latest full 393798 -18475> 2013-04-03 10:36:40.745745 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394008 update_from_paxos applying incremental log 394008 2013-04-03 10:36:39.664253 mon.0 10.252.24.25:6789/0 65 : [INF] pgmap v29003: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18474> 2013-04-03 10:36:40.745790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 check_subs -18473> 2013-04-03 10:36:40.745827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 create_pending v 394009 -18472> 2013-04-03 10:36:40.745850 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394008 _updated_log for mon.0 10.252.24.25:6789/0 -18471> 2013-04-03 10:36:40.745862 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 65) v1 -- ?+0 0x2294380 con 0x1bac160 -18470> 2013-04-03 10:36:40.746157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos -18469> 2013-04-03 10:36:40.746175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394008 -18468> 2013-04-03 10:36:40.746183 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18467> 2013-04-03 10:36:40.746194 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 preprocess_log log(1 entries) v1 from mon.0 -18466> 2013-04-03 10:36:40.746210 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 nothing new -18465> 2013-04-03 10:36:40.746258 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos -18464> 2013-04-03 10:36:40.746273 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394008 -18463> 2013-04-03 10:36:40.746289 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18462> 2013-04-03 10:36:40.746307 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 65) v1 ==== 0+0+0 (0 0 0) 0x2294380 con 0x1bac160 -18461> 2013-04-03 10:36:40.746313 7fcdffcdb700 10 handle_log_ack log(last 65) v1 -18460> 2013-04-03 10:36:40.746314 7fcdffcdb700 10 logged 2013-04-03 10:36:39.664253 mon.0 10.252.24.25:6789/0 65 : [INF] pgmap v29003: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -18459> 2013-04-03 10:36:40.758677 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113195 ==== paxos(lease_ack lc 427291 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (3077679545 0 0) 0x29fda00 con 0x208a840 -18458> 2013-04-03 10:36:40.758712 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos -18457> 2013-04-03 10:36:40.758728 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394008 -18456> 2013-04-03 10:36:40.758742 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18455> 2013-04-03 10:36:40.926423 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113196 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 48 v187) v2) to leader v1 ==== 560+0+0 (3068235052 0 0) 0x29f4b00 con 0x208a840 -18454> 2013-04-03 10:36:40.926437 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -18453> 2013-04-03 10:36:40.926445 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29f4840 from 10.252.24.88:6789/0 -18452> 2013-04-03 10:36:40.926476 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 48 v187) v2 from mds.0 10.252.24.25:6802/3144 -18451> 2013-04-03 10:36:40.926486 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 48 v187) v2 tid 17479) v2 -- ?+0 0x20fc240 con 0x208a840 -18450> 2013-04-03 10:36:41.380330 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 46 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21704 v187) v2 ==== 289+0+0 (2086745478 0 0) 0x295e340 con 0x200d420 -18449> 2013-04-03 10:36:41.380401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21704 v187) v2 from mds.? 10.252.24.88:6800/17213 -18448> 2013-04-03 10:36:41.380414 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21704 v187) v2 -- ?+0 0x29f4b00 con 0x200d420 -18447> 2013-04-03 10:36:43.332720 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29004 check_down_pgs -18446> 2013-04-03 10:36:43.332747 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29004 v29004: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 422B/s wr, 0op/s -18445> 2013-04-03 10:36:43.332774 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -18444> 2013-04-03 10:36:43.332796 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -18443> 2013-04-03 10:36:43.332832 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -18442> 2013-04-03 10:36:43.332862 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos -18441> 2013-04-03 10:36:43.332881 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394008 -18440> 2013-04-03 10:36:43.332890 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394008 log -18439> 2013-04-03 10:36:43.332906 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18438> 2013-04-03 10:36:43.332925 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -18437> 2013-04-03 10:36:43.491425 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113197 ==== forward(pg_stats(1 pgs tid 10 v 602) v1) to leader v1 ==== 757+0+0 (1904758177 0 0) 0x29f4dc0 con 0x208a840 -18436> 2013-04-03 10:36:43.491447 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 -18435> 2013-04-03 10:36:43.491456 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20fc240 from 10.252.24.88:6789/0 -18434> 2013-04-03 10:36:43.491496 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 preprocess_query pg_stats(1 pgs tid 10 v 602) v1 from osd.2 10.252.24.30:6800/1508 -18433> 2013-04-03 10:36:43.491518 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 prepare_update pg_stats(1 pgs tid 10 v 602) v1 from osd.2 10.252.24.30:6800/1508 -18432> 2013-04-03 10:36:43.491524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 prepare_pg_stats pg_stats(1 pgs tid 10 v 602) v1 from osd.2 -18431> 2013-04-03 10:36:43.491530 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29004 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -18430> 2013-04-03 10:36:43.541707 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29004 encode_pending v 29005 -18429> 2013-04-03 10:36:43.541737 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427291) queue_proposal bl 654 bytes; ctx = 0x1b91dc0 -18428> 2013-04-03 10:36:43.541745 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427291) propose_queued 427292 654 bytes -18427> 2013-04-03 10:36:43.543290 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427291 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd780 -18426> 2013-04-03 10:36:43.572378 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113198 ==== paxos(accept lc 427291 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3904193344 0 0) 0x29fd780 con 0x208a840 -18425> 2013-04-03 10:36:43.572716 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427292 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fda00 -18424> 2013-04-03 10:36:43.572745 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427292 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29fd500 -18423> 2013-04-03 10:36:43.572816 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29004 update_from_paxos applying incremental 29005 -18422> 2013-04-03 10:36:43.572839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 v29005: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 4438B/s wr, 0op/s -18421> 2013-04-03 10:36:43.572857 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 send_pg_creates to 0 pgs -18420> 2013-04-03 10:36:43.572861 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 update_logger -18419> 2013-04-03 10:36:43.572872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 create_pending v 29006 -18418> 2013-04-03 10:36:43.572880 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29005 _updated_stats for osd.2 10.252.24.30:6800/1508 -18417> 2013-04-03 10:36:43.572886 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 10) v1 tid 17480) v2 -- ?+0 0x20adb40 con 0x208a840 -18416> 2013-04-03 10:36:43.573090 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 check_osd_map already seen 602 >= 602 -18415> 2013-04-03 10:36:43.573094 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 update_logger -18414> 2013-04-03 10:36:43.573105 7fcdffcdb700 0 log [INF] : pgmap v29005: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 4438B/s wr, 0op/s -18413> 2013-04-03 10:36:43.573107 7fcdffcdb700 10 send_log to self -18412> 2013-04-03 10:36:43.573107 7fcdffcdb700 10 log_queue is 2 last_log 67 sent 65 num 2 unsent 2 sending 2 -18411> 2013-04-03 10:36:43.573109 7fcdffcdb700 10 will send 2013-04-03 10:36:40.708773 mon.0 10.252.24.25:6789/0 66 : [INF] pgmap v29004: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 422B/s wr, 0op/s -18410> 2013-04-03 10:36:43.573116 7fcdffcdb700 10 will send 2013-04-03 10:36:43.573106 mon.0 10.252.24.25:6789/0 67 : [INF] pgmap v29005: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 4438B/s wr, 0op/s -18409> 2013-04-03 10:36:43.573120 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x287ee00 -18408> 2013-04-03 10:36:43.573167 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos -18407> 2013-04-03 10:36:43.573184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394008 -18406> 2013-04-03 10:36:43.573202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18405> 2013-04-03 10:36:43.573224 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x287ee00 con 0x1bac160 -18404> 2013-04-03 10:36:43.573244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos -18403> 2013-04-03 10:36:43.573263 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394008 -18402> 2013-04-03 10:36:43.573272 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -18401> 2013-04-03 10:36:43.573282 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 preprocess_log log(2 entries) v1 from mon.0 -18400> 2013-04-03 10:36:43.573308 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -18399> 2013-04-03 10:36:43.573320 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 prepare_log log(2 entries) v1 from mon.0 -18398> 2013-04-03 10:36:43.573330 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 logging 2013-04-03 10:36:40.708773 mon.0 10.252.24.25:6789/0 66 : [INF] pgmap v29004: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 422B/s wr, 0op/s -18397> 2013-04-03 10:36:43.573350 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 logging 2013-04-03 10:36:43.573106 mon.0 10.252.24.25:6789/0 67 : [INF] pgmap v29005: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 4438B/s wr, 0op/s -18396> 2013-04-03 10:36:43.580601 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 47 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140334 v187) v2 ==== 289+0+0 (4211436041 0 0) 0x2995dc0 con 0x2118420 -18395> 2013-04-03 10:36:43.580660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140334 v187) v2 from mds.? 10.252.58.31:6800/10908 -18394> 2013-04-03 10:36:43.580671 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140334 v187) v2 -- ?+0 0x29f4dc0 con 0x2118420 -18393> 2013-04-03 10:36:43.597630 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113199 ==== paxos(lease_ack lc 427292 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (1822635388 0 0) 0x29fd500 con 0x208a840 -18392> 2013-04-03 10:36:43.597677 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos -18391> 2013-04-03 10:36:43.597694 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394008 update_from_paxos version 394008 summary v 394008 -18390> 2013-04-03 10:36:43.597709 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18389> 2013-04-03 10:36:44.572579 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394008 encode_pending v394009 -18388> 2013-04-03 10:36:44.572616 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427292) queue_proposal bl 725 bytes; ctx = 0x1b91e90 -18387> 2013-04-03 10:36:44.572624 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427292) propose_queued 427293 725 bytes -18386> 2013-04-03 10:36:44.573844 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427292 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0780 -18385> 2013-04-03 10:36:44.596796 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113200 ==== paxos(accept lc 427292 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3985592440 0 0) 0x29e0780 con 0x208a840 -18384> 2013-04-03 10:36:44.597059 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427293 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd500 -18383> 2013-04-03 10:36:44.597089 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427293 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29fd780 -18382> 2013-04-03 10:36:44.597130 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos -18381> 2013-04-03 10:36:44.597156 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394008 -18380> 2013-04-03 10:36:44.597174 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos latest full 393798 -18379> 2013-04-03 10:36:44.597190 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394009 update_from_paxos applying incremental log 394009 2013-04-03 10:36:40.708773 mon.0 10.252.24.25:6789/0 66 : [INF] pgmap v29004: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 422B/s wr, 0op/s -18378> 2013-04-03 10:36:44.597217 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394009 update_from_paxos applying incremental log 394009 2013-04-03 10:36:43.573106 mon.0 10.252.24.25:6789/0 67 : [INF] pgmap v29005: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 4438B/s wr, 0op/s -18377> 2013-04-03 10:36:44.597256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 check_subs -18376> 2013-04-03 10:36:44.597286 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 create_pending v 394010 -18375> 2013-04-03 10:36:44.597308 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394009 _updated_log for mon.0 10.252.24.25:6789/0 -18374> 2013-04-03 10:36:44.597320 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 67) v1 -- ?+0 0x2294700 con 0x1bac160 -18373> 2013-04-03 10:36:44.597371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos -18372> 2013-04-03 10:36:44.597387 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394009 -18371> 2013-04-03 10:36:44.597403 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18370> 2013-04-03 10:36:44.597423 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 67) v1 ==== 0+0+0 (0 0 0) 0x2294700 con 0x1bac160 -18369> 2013-04-03 10:36:44.597431 7fcdffcdb700 10 handle_log_ack log(last 67) v1 -18368> 2013-04-03 10:36:44.597433 7fcdffcdb700 10 logged 2013-04-03 10:36:40.708773 mon.0 10.252.24.25:6789/0 66 : [INF] pgmap v29004: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 422B/s wr, 0op/s -18367> 2013-04-03 10:36:44.597439 7fcdffcdb700 10 logged 2013-04-03 10:36:43.573106 mon.0 10.252.24.25:6789/0 67 : [INF] pgmap v29005: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 4438B/s wr, 0op/s -18366> 2013-04-03 10:36:44.608718 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113201 ==== paxos(lease_ack lc 427293 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (2573137196 0 0) 0x29fd780 con 0x208a840 -18365> 2013-04-03 10:36:44.608753 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos -18364> 2013-04-03 10:36:44.608769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394009 -18363> 2013-04-03 10:36:44.608783 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18362> 2013-04-03 10:36:44.926564 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113202 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 49 v187) v2) to leader v1 ==== 560+0+0 (894664242 0 0) 0x29f4580 con 0x208a840 -18361> 2013-04-03 10:36:44.926581 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -18360> 2013-04-03 10:36:44.926590 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29f42c0 from 10.252.24.88:6789/0 -18359> 2013-04-03 10:36:44.926622 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 49 v187) v2 from mds.0 10.252.24.25:6802/3144 -18358> 2013-04-03 10:36:44.926633 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 49 v187) v2 tid 17481) v2 -- ?+0 0x24f8900 con 0x208a840 -18357> 2013-04-03 10:36:45.380406 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 47 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21705 v187) v2 ==== 289+0+0 (1594854139 0 0) 0x295d580 con 0x200d420 -18356> 2013-04-03 10:36:45.380477 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21705 v187) v2 from mds.? 10.252.24.88:6800/17213 -18355> 2013-04-03 10:36:45.380489 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21705 v187) v2 -- ?+0 0x29f4580 con 0x200d420 -18354> 2013-04-03 10:36:47.580744 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 48 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140335 v187) v2 ==== 289+0+0 (3630945652 0 0) 0x2996340 con 0x2118420 -18353> 2013-04-03 10:36:47.580817 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140335 v187) v2 from mds.? 10.252.58.31:6800/10908 -18352> 2013-04-03 10:36:47.580829 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140335 v187) v2 -- ?+0 0x295d580 con 0x2118420 -18351> 2013-04-03 10:36:47.597161 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427293 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29e0c80 -18350> 2013-04-03 10:36:47.597957 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113203 ==== paxos(lease_ack lc 427293 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (68485675 0 0) 0x29e0c80 con 0x208a840 -18349> 2013-04-03 10:36:47.598008 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos -18348> 2013-04-03 10:36:47.598028 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394009 -18347> 2013-04-03 10:36:47.598044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18346> 2013-04-03 10:36:48.333076 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29005 check_down_pgs -18345> 2013-04-03 10:36:48.333107 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29005 v29005: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 4438B/s wr, 0op/s -18344> 2013-04-03 10:36:48.333134 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -18343> 2013-04-03 10:36:48.333156 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -18342> 2013-04-03 10:36:48.333193 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -18341> 2013-04-03 10:36:48.333223 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos -18340> 2013-04-03 10:36:48.333242 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394009 -18339> 2013-04-03 10:36:48.333251 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394009 log -18338> 2013-04-03 10:36:48.333268 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18337> 2013-04-03 10:36:48.333286 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -18336> 2013-04-03 10:36:48.431186 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113204 ==== forward(pg_stats(0 pgs tid 58 v 602) v1) to leader v1 ==== 368+0+0 (3927864539 0 0) 0x2995dc0 con 0x208a840 -18335> 2013-04-03 10:36:48.431214 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 -18334> 2013-04-03 10:36:48.431224 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -18333> 2013-04-03 10:36:48.431274 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 preprocess_query pg_stats(0 pgs tid 58 v 602) v1 from osd.4 10.252.24.32:6800/13473 -18332> 2013-04-03 10:36:48.431297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 prepare_update pg_stats(0 pgs tid 58 v 602) v1 from osd.4 10.252.24.32:6800/13473 -18331> 2013-04-03 10:36:48.431308 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 prepare_pg_stats pg_stats(0 pgs tid 58 v 602) v1 from osd.4 -18330> 2013-04-03 10:36:48.431314 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29005 got osd.4 osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1051 MB used, 56606 MB avail, 59728 MB total, peers []/[])) -18329> 2013-04-03 10:36:48.481461 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29005 encode_pending v 29006 -18328> 2013-04-03 10:36:48.481496 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427293) queue_proposal bl 265 bytes; ctx = 0x1b91c60 -18327> 2013-04-03 10:36:48.481505 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427293) propose_queued 427294 265 bytes -18326> 2013-04-03 10:36:48.482798 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427293 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0a00 -18325> 2013-04-03 10:36:48.514095 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113205 ==== paxos(accept lc 427293 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3241074439 0 0) 0x29e0a00 con 0x208a840 -18324> 2013-04-03 10:36:48.522530 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427294 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0c80 -18323> 2013-04-03 10:36:48.522560 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427294 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29fd780 -18322> 2013-04-03 10:36:48.522636 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29005 update_from_paxos applying incremental 29006 -18321> 2013-04-03 10:36:48.522646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 v29006: 240 pgs: 240 active+clean; 253 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 2095B/s wr, 0op/s -18320> 2013-04-03 10:36:48.522662 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 send_pg_creates to 0 pgs -18319> 2013-04-03 10:36:48.522664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 update_logger -18318> 2013-04-03 10:36:48.522673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 create_pending v 29007 -18317> 2013-04-03 10:36:48.522682 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29006 _updated_stats for osd.4 10.252.24.32:6800/13473 -18316> 2013-04-03 10:36:48.522687 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(0 pgs tid 58) v1 tid 17482) v2 -- ?+0 0x20adb40 con 0x208a840 -18315> 2013-04-03 10:36:48.522697 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 check_osd_map already seen 602 >= 602 -18314> 2013-04-03 10:36:48.522699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 update_logger -18313> 2013-04-03 10:36:48.522709 7fcdffcdb700 0 log [INF] : pgmap v29006: 240 pgs: 240 active+clean; 253 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 2095B/s wr, 0op/s -18312> 2013-04-03 10:36:48.522710 7fcdffcdb700 10 send_log to self -18311> 2013-04-03 10:36:48.522711 7fcdffcdb700 10 log_queue is 1 last_log 68 sent 67 num 1 unsent 1 sending 1 -18310> 2013-04-03 10:36:48.522713 7fcdffcdb700 10 will send 2013-04-03 10:36:48.522710 mon.0 10.252.24.25:6789/0 68 : [INF] pgmap v29006: 240 pgs: 240 active+clean; 253 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 2095B/s wr, 0op/s -18309> 2013-04-03 10:36:48.522722 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287ee00 -18308> 2013-04-03 10:36:48.522776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos -18307> 2013-04-03 10:36:48.522796 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394009 -18306> 2013-04-03 10:36:48.522814 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18305> 2013-04-03 10:36:48.522835 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287ee00 con 0x1bac160 -18304> 2013-04-03 10:36:48.522857 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos -18303> 2013-04-03 10:36:48.522872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394009 -18302> 2013-04-03 10:36:48.522881 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18301> 2013-04-03 10:36:48.522891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 preprocess_log log(1 entries) v1 from mon.0 -18300> 2013-04-03 10:36:48.522911 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18299> 2013-04-03 10:36:48.522923 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 prepare_log log(1 entries) v1 from mon.0 -18298> 2013-04-03 10:36:48.522933 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 logging 2013-04-03 10:36:48.522710 mon.0 10.252.24.25:6789/0 68 : [INF] pgmap v29006: 240 pgs: 240 active+clean; 253 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 2095B/s wr, 0op/s -18297> 2013-04-03 10:36:48.534330 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113206 ==== paxos(lease_ack lc 427294 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (843969608 0 0) 0x29fd780 con 0x208a840 -18296> 2013-04-03 10:36:48.534373 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos -18295> 2013-04-03 10:36:48.534390 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394009 update_from_paxos version 394009 summary v 394009 -18294> 2013-04-03 10:36:48.534404 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18293> 2013-04-03 10:36:48.926707 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113207 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 50 v187) v2) to leader v1 ==== 560+0+0 (1654478085 0 0) 0x29f4000 con 0x208a840 -18292> 2013-04-03 10:36:48.926724 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -18291> 2013-04-03 10:36:48.926732 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x295d580 from 10.252.24.88:6789/0 -18290> 2013-04-03 10:36:48.926766 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 50 v187) v2 from mds.0 10.252.24.25:6802/3144 -18289> 2013-04-03 10:36:48.926778 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 50 v187) v2 tid 17483) v2 -- ?+0 0x24f8900 con 0x208a840 -18288> 2013-04-03 10:36:49.380549 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 48 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21706 v187) v2 ==== 289+0+0 (985418620 0 0) 0x2145000 con 0x200d420 -18287> 2013-04-03 10:36:49.380627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21706 v187) v2 from mds.? 10.252.24.88:6800/17213 -18286> 2013-04-03 10:36:49.380640 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21706 v187) v2 -- ?+0 0x29f4000 con 0x200d420 -18285> 2013-04-03 10:36:49.514286 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394009 encode_pending v394010 -18284> 2013-04-03 10:36:49.514312 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427294) queue_proposal bl 443 bytes; ctx = 0x1b91e80 -18283> 2013-04-03 10:36:49.514319 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427294) propose_queued 427295 443 bytes -18282> 2013-04-03 10:36:49.515469 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427294 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1a00 -18281> 2013-04-03 10:36:49.541865 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113208 ==== paxos(accept lc 427294 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4167169237 0 0) 0x29a1a00 con 0x208a840 -18280> 2013-04-03 10:36:49.542155 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427295 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd780 -18279> 2013-04-03 10:36:49.542175 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427295 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29e0a00 -18278> 2013-04-03 10:36:49.542220 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos -18277> 2013-04-03 10:36:49.542246 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos version 394010 summary v 394009 -18276> 2013-04-03 10:36:49.542260 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos latest full 393798 -18275> 2013-04-03 10:36:49.542276 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394010 update_from_paxos applying incremental log 394010 2013-04-03 10:36:48.522710 mon.0 10.252.24.25:6789/0 68 : [INF] pgmap v29006: 240 pgs: 240 active+clean; 253 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 2095B/s wr, 0op/s -18274> 2013-04-03 10:36:49.542323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 check_subs -18273> 2013-04-03 10:36:49.542354 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 create_pending v 394011 -18272> 2013-04-03 10:36:49.542375 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394010 _updated_log for mon.0 10.252.24.25:6789/0 -18271> 2013-04-03 10:36:49.542388 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 68) v1 -- ?+0 0x2294a80 con 0x1bac160 -18270> 2013-04-03 10:36:49.542437 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos -18269> 2013-04-03 10:36:49.542452 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos version 394010 summary v 394010 -18268> 2013-04-03 10:36:49.542470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18267> 2013-04-03 10:36:49.542491 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 68) v1 ==== 0+0+0 (0 0 0) 0x2294a80 con 0x1bac160 -18266> 2013-04-03 10:36:49.542502 7fcdffcdb700 10 handle_log_ack log(last 68) v1 -18265> 2013-04-03 10:36:49.542504 7fcdffcdb700 10 logged 2013-04-03 10:36:48.522710 mon.0 10.252.24.25:6789/0 68 : [INF] pgmap v29006: 240 pgs: 240 active+clean; 253 MB data, 5653 MB used, 1067 GB / 1079 GB avail; 2095B/s wr, 0op/s -18264> 2013-04-03 10:36:49.553847 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113209 ==== paxos(lease_ack lc 427295 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (111504147 0 0) 0x29e0a00 con 0x208a840 -18263> 2013-04-03 10:36:49.553883 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos -18262> 2013-04-03 10:36:49.553899 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos version 394010 summary v 394010 -18261> 2013-04-03 10:36:49.553913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18260> 2013-04-03 10:36:49.800634 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113210 ==== forward(pg_stats(0 pgs tid 407 v 602) v1) to leader v1 ==== 384+0+0 (2012211438 0 0) 0x2995dc0 con 0x208a840 -18259> 2013-04-03 10:36:49.800649 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -18258> 2013-04-03 10:36:49.800658 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -18257> 2013-04-03 10:36:49.800687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 preprocess_query pg_stats(0 pgs tid 407 v 602) v1 from osd.0 10.252.58.31:6802/2682 -18256> 2013-04-03 10:36:49.800708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 prepare_update pg_stats(0 pgs tid 407 v 602) v1 from osd.0 10.252.58.31:6802/2682 -18255> 2013-04-03 10:36:49.800713 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 prepare_pg_stats pg_stats(0 pgs tid 407 v 602) v1 from osd.0 -18254> 2013-04-03 10:36:49.800719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29006 got osd.0 osd_stat(1139 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1138 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -18253> 2013-04-03 10:36:50.329264 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x29b4280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:32985/0 -18252> 2013-04-03 10:36:50.329299 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:32985/0) -18251> 2013-04-03 10:36:50.329322 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4280 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -18250> 2013-04-03 10:36:50.329569 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4280 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -18249> 2013-04-03 10:36:50.542071 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29006 encode_pending v 29007 -18248> 2013-04-03 10:36:50.542115 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427295) queue_proposal bl 281 bytes; ctx = 0x1b91c70 -18247> 2013-04-03 10:36:50.542125 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427295) propose_queued 427296 281 bytes -18246> 2013-04-03 10:36:50.543526 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427295 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1780 -18245> 2013-04-03 10:36:50.711412 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113211 ==== paxos(accept lc 427295 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (994045205 0 0) 0x29a1780 con 0x208a840 -18244> 2013-04-03 10:36:50.711799 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427296 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0a00 -18243> 2013-04-03 10:36:50.711828 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427296 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29a1a00 -18242> 2013-04-03 10:36:50.711917 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29006 update_from_paxos applying incremental 29007 -18241> 2013-04-03 10:36:50.711929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 v29007: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -18240> 2013-04-03 10:36:50.711946 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 send_pg_creates to 0 pgs -18239> 2013-04-03 10:36:50.711950 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 update_logger -18238> 2013-04-03 10:36:50.711960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 create_pending v 29008 -18237> 2013-04-03 10:36:50.711968 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29007 _updated_stats for osd.0 10.252.58.31:6802/2682 -18236> 2013-04-03 10:36:50.711974 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(0 pgs tid 407) v1 tid 17484) v2 -- ?+0 0x20adb40 con 0x208a840 -18235> 2013-04-03 10:36:50.711985 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 check_osd_map already seen 602 >= 602 -18234> 2013-04-03 10:36:50.711987 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 update_logger -18233> 2013-04-03 10:36:50.711997 7fcdffcdb700 0 log [INF] : pgmap v29007: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -18232> 2013-04-03 10:36:50.711998 7fcdffcdb700 10 send_log to self -18231> 2013-04-03 10:36:50.711999 7fcdffcdb700 10 log_queue is 1 last_log 69 sent 68 num 1 unsent 1 sending 1 -18230> 2013-04-03 10:36:50.712000 7fcdffcdb700 10 will send 2013-04-03 10:36:50.711998 mon.0 10.252.24.25:6789/0 69 : [INF] pgmap v29007: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -18229> 2013-04-03 10:36:50.712007 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287ee00 -18228> 2013-04-03 10:36:50.712063 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos -18227> 2013-04-03 10:36:50.712083 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos version 394010 summary v 394010 -18226> 2013-04-03 10:36:50.712099 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18225> 2013-04-03 10:36:50.712120 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287ee00 con 0x1bac160 -18224> 2013-04-03 10:36:50.712143 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos -18223> 2013-04-03 10:36:50.712158 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos version 394010 summary v 394010 -18222> 2013-04-03 10:36:50.712165 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18221> 2013-04-03 10:36:50.712175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 preprocess_log log(1 entries) v1 from mon.0 -18220> 2013-04-03 10:36:50.712195 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18219> 2013-04-03 10:36:50.712207 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 prepare_log log(1 entries) v1 from mon.0 -18218> 2013-04-03 10:36:50.712216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 logging 2013-04-03 10:36:50.711998 mon.0 10.252.24.25:6789/0 69 : [INF] pgmap v29007: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -18217> 2013-04-03 10:36:50.724554 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113212 ==== paxos(lease_ack lc 427296 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (2165198377 0 0) 0x29a1a00 con 0x208a840 -18216> 2013-04-03 10:36:50.724588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos -18215> 2013-04-03 10:36:50.724603 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394010 update_from_paxos version 394010 summary v 394010 -18214> 2013-04-03 10:36:50.724617 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18213> 2013-04-03 10:36:51.580874 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 49 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140336 v187) v2 ==== 289+0+0 (331906604 0 0) 0x2a52b80 con 0x2118420 -18212> 2013-04-03 10:36:51.580950 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140336 v187) v2 from mds.? 10.252.58.31:6800/10908 -18211> 2013-04-03 10:36:51.580963 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140336 v187) v2 -- ?+0 0x2995dc0 con 0x2118420 -18210> 2013-04-03 10:36:51.711958 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394010 encode_pending v394011 -18209> 2013-04-03 10:36:51.711990 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427296) queue_proposal bl 424 bytes; ctx = 0x1b91c90 -18208> 2013-04-03 10:36:51.711999 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427296) propose_queued 427297 424 bytes -18207> 2013-04-03 10:36:51.713235 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427296 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2981000 -18206> 2013-04-03 10:36:51.740325 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113213 ==== paxos(accept lc 427296 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1079104679 0 0) 0x2981000 con 0x208a840 -18205> 2013-04-03 10:36:51.740624 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427297 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1a00 -18204> 2013-04-03 10:36:51.740655 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427297 fc 427258 pn 0 opn 0) v3 -- ?+0 0x29a1780 -18203> 2013-04-03 10:36:51.740695 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos -18202> 2013-04-03 10:36:51.740721 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos version 394011 summary v 394010 -18201> 2013-04-03 10:36:51.740738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos latest full 393798 -18200> 2013-04-03 10:36:51.740757 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394011 update_from_paxos applying incremental log 394011 2013-04-03 10:36:50.711998 mon.0 10.252.24.25:6789/0 69 : [INF] pgmap v29007: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -18199> 2013-04-03 10:36:51.740800 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 check_subs -18198> 2013-04-03 10:36:51.740830 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 create_pending v 394012 -18197> 2013-04-03 10:36:51.740852 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394011 _updated_log for mon.0 10.252.24.25:6789/0 -18196> 2013-04-03 10:36:51.740865 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 69) v1 -- ?+0 0x2294540 con 0x1bac160 -18195> 2013-04-03 10:36:51.740912 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos -18194> 2013-04-03 10:36:51.740927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos version 394011 summary v 394011 -18193> 2013-04-03 10:36:51.740945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18192> 2013-04-03 10:36:51.740965 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 69) v1 ==== 0+0+0 (0 0 0) 0x2294540 con 0x1bac160 -18191> 2013-04-03 10:36:51.740973 7fcdffcdb700 10 handle_log_ack log(last 69) v1 -18190> 2013-04-03 10:36:51.740975 7fcdffcdb700 10 logged 2013-04-03 10:36:50.711998 mon.0 10.252.24.25:6789/0 69 : [INF] pgmap v29007: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -18189> 2013-04-03 10:36:51.752322 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113214 ==== paxos(lease_ack lc 427297 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (2740701210 0 0) 0x29a1780 con 0x208a840 -18188> 2013-04-03 10:36:51.752357 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos -18187> 2013-04-03 10:36:51.752373 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos version 394011 summary v 394011 -18186> 2013-04-03 10:36:51.752387 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18185> 2013-04-03 10:36:52.926850 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113215 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 51 v187) v2) to leader v1 ==== 560+0+0 (2320826985 0 0) 0x29f4dc0 con 0x208a840 -18184> 2013-04-03 10:36:52.926871 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -18183> 2013-04-03 10:36:52.926881 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c5b80 from 10.252.24.88:6789/0 -18182> 2013-04-03 10:36:52.926927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 51 v187) v2 from mds.0 10.252.24.25:6802/3144 -18181> 2013-04-03 10:36:52.926939 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 51 v187) v2 tid 17485) v2 -- ?+0 0x24f8900 con 0x208a840 -18180> 2013-04-03 10:36:53.333433 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29007 check_down_pgs -18179> 2013-04-03 10:36:53.333459 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29007 v29007: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -18178> 2013-04-03 10:36:53.333484 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -18177> 2013-04-03 10:36:53.333507 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -18176> 2013-04-03 10:36:53.333544 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -18175> 2013-04-03 10:36:53.333574 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos -18174> 2013-04-03 10:36:53.333592 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos version 394011 summary v 394011 -18173> 2013-04-03 10:36:53.333602 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394011 log -18172> 2013-04-03 10:36:53.333621 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18171> 2013-04-03 10:36:53.333639 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -18170> 2013-04-03 10:36:53.380629 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 49 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21707 v187) v2 ==== 289+0+0 (433255425 0 0) 0x20e4340 con 0x200d420 -18169> 2013-04-03 10:36:53.380674 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21707 v187) v2 from mds.? 10.252.24.88:6800/17213 -18168> 2013-04-03 10:36:53.380684 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21707 v187) v2 -- ?+0 0x29f4dc0 con 0x200d420 -18167> 2013-04-03 10:36:54.570882 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113216 ==== forward(pg_stats(1 pgs tid 18 v 602) v1) to leader v1 ==== 757+0+0 (1568359333 0 0) 0x2a52b80 con 0x208a840 -18166> 2013-04-03 10:36:54.570902 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -18165> 2013-04-03 10:36:54.570911 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -18164> 2013-04-03 10:36:54.570961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 preprocess_query pg_stats(1 pgs tid 18 v 602) v1 from osd.1 10.252.24.25:6805/3572 -18163> 2013-04-03 10:36:54.570984 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 prepare_update pg_stats(1 pgs tid 18 v 602) v1 from osd.1 10.252.24.25:6805/3572 -18162> 2013-04-03 10:36:54.570990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 prepare_pg_stats pg_stats(1 pgs tid 18 v 602) v1 from osd.1 -18161> 2013-04-03 10:36:54.570997 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29007 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -18160> 2013-04-03 10:36:54.621178 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29007 encode_pending v 29008 -18159> 2013-04-03 10:36:54.621214 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427297) queue_proposal bl 654 bytes; ctx = 0x1b91d20 -18158> 2013-04-03 10:36:54.621224 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427297) propose_queued 427298 654 bytes -18157> 2013-04-03 10:36:54.622158 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427297 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2981a00 -18156> 2013-04-03 10:36:54.649333 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113217 ==== paxos(accept lc 427297 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2661990403 0 0) 0x2981a00 con 0x208a840 -18155> 2013-04-03 10:36:54.655655 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427298 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1780 -18154> 2013-04-03 10:36:54.655688 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427298 fc 427258 pn 0 opn 0) v3 -- ?+0 0x2981000 -18153> 2013-04-03 10:36:54.655759 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29007 update_from_paxos applying incremental 29008 -18152> 2013-04-03 10:36:54.655774 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 v29008: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 667B/s wr, 0op/s -18151> 2013-04-03 10:36:54.655792 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 send_pg_creates to 0 pgs -18150> 2013-04-03 10:36:54.655796 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 update_logger -18149> 2013-04-03 10:36:54.655809 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 create_pending v 29009 -18148> 2013-04-03 10:36:54.655818 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29008 _updated_stats for osd.1 10.252.24.25:6805/3572 -18147> 2013-04-03 10:36:54.655823 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 18) v1 tid 17486) v2 -- ?+0 0x20adb40 con 0x208a840 -18146> 2013-04-03 10:36:54.655833 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 check_osd_map already seen 602 >= 602 -18145> 2013-04-03 10:36:54.655835 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 update_logger -18144> 2013-04-03 10:36:54.655845 7fcdffcdb700 0 log [INF] : pgmap v29008: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 667B/s wr, 0op/s -18143> 2013-04-03 10:36:54.655846 7fcdffcdb700 10 send_log to self -18142> 2013-04-03 10:36:54.655847 7fcdffcdb700 10 log_queue is 1 last_log 70 sent 69 num 1 unsent 1 sending 1 -18141> 2013-04-03 10:36:54.655849 7fcdffcdb700 10 will send 2013-04-03 10:36:54.655846 mon.0 10.252.24.25:6789/0 70 : [INF] pgmap v29008: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 667B/s wr, 0op/s -18140> 2013-04-03 10:36:54.655855 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287ee00 -18139> 2013-04-03 10:36:54.655901 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos -18138> 2013-04-03 10:36:54.655919 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos version 394011 summary v 394011 -18137> 2013-04-03 10:36:54.655937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18136> 2013-04-03 10:36:54.655959 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287ee00 con 0x1bac160 -18135> 2013-04-03 10:36:54.655980 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos -18134> 2013-04-03 10:36:54.655994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos version 394011 summary v 394011 -18133> 2013-04-03 10:36:54.656003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18132> 2013-04-03 10:36:54.656014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 preprocess_log log(1 entries) v1 from mon.0 -18131> 2013-04-03 10:36:54.656034 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18130> 2013-04-03 10:36:54.656049 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 prepare_log log(1 entries) v1 from mon.0 -18129> 2013-04-03 10:36:54.656058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 logging 2013-04-03 10:36:54.655846 mon.0 10.252.24.25:6789/0 70 : [INF] pgmap v29008: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 667B/s wr, 0op/s -18128> 2013-04-03 10:36:54.670936 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113218 ==== paxos(lease_ack lc 427298 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (2291226666 0 0) 0x2981000 con 0x208a840 -18127> 2013-04-03 10:36:54.670971 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos -18126> 2013-04-03 10:36:54.670986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394011 update_from_paxos version 394011 summary v 394011 -18125> 2013-04-03 10:36:54.671000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18124> 2013-04-03 10:36:55.580964 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 50 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140337 v187) v2 ==== 289+0+0 (816234833 0 0) 0x2a528c0 con 0x2118420 -18123> 2013-04-03 10:36:55.581039 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140337 v187) v2 from mds.? 10.252.58.31:6800/10908 -18122> 2013-04-03 10:36:55.581051 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140337 v187) v2 -- ?+0 0x2a52b80 con 0x2118420 -18121> 2013-04-03 10:36:55.649517 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394011 encode_pending v394012 -18120> 2013-04-03 10:36:55.649544 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427298) queue_proposal bl 442 bytes; ctx = 0x1b91da0 -18119> 2013-04-03 10:36:55.649551 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427298) propose_queued 427299 442 bytes -18118> 2013-04-03 10:36:55.650802 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427298 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909a00 -18117> 2013-04-03 10:36:55.853308 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113219 ==== paxos(accept lc 427298 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3794317491 0 0) 0x2909a00 con 0x208a840 -18116> 2013-04-03 10:36:55.853615 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427299 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2981000 -18115> 2013-04-03 10:36:55.853649 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427299 fc 427258 pn 0 opn 0) v3 -- ?+0 0x2981a00 -18114> 2013-04-03 10:36:55.853686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos -18113> 2013-04-03 10:36:55.853712 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394011 -18112> 2013-04-03 10:36:55.853726 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos latest full 393798 -18111> 2013-04-03 10:36:55.853742 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394012 update_from_paxos applying incremental log 394012 2013-04-03 10:36:54.655846 mon.0 10.252.24.25:6789/0 70 : [INF] pgmap v29008: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 667B/s wr, 0op/s -18110> 2013-04-03 10:36:55.853788 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 check_subs -18109> 2013-04-03 10:36:55.853826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 create_pending v 394013 -18108> 2013-04-03 10:36:55.853849 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394012 _updated_log for mon.0 10.252.24.25:6789/0 -18107> 2013-04-03 10:36:55.853864 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 70) v1 -- ?+0 0x2294e00 con 0x1bac160 -18106> 2013-04-03 10:36:55.853938 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427258..427299) queue_proposal bl 987 bytes; ctx = 0x1b91d50 -18105> 2013-04-03 10:36:55.853947 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427258..427299) propose_queued 427300 987 bytes -18104> 2013-04-03 10:36:55.855333 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427299 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0780 -18103> 2013-04-03 10:36:55.855368 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 70) v1 ==== 0+0+0 (0 0 0) 0x2294e00 con 0x1bac160 -18102> 2013-04-03 10:36:55.855385 7fcdffcdb700 10 handle_log_ack log(last 70) v1 -18101> 2013-04-03 10:36:55.855386 7fcdffcdb700 10 logged 2013-04-03 10:36:54.655846 mon.0 10.252.24.25:6789/0 70 : [INF] pgmap v29008: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 667B/s wr, 0op/s -18100> 2013-04-03 10:36:55.865176 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113220 ==== paxos(lease_ack lc 427299 fc 427258 pn 0 opn 0) v3 ==== 80+0+0 (3405128225 0 0) 0x29e0780 con 0x208a840 -18099> 2013-04-03 10:36:55.890379 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113221 ==== paxos(accept lc 427299 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3880831695 0 0) 0x2981a00 con 0x208a840 -18098> 2013-04-03 10:36:55.890717 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427300 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0780 -18097> 2013-04-03 10:36:55.890738 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427300 fc 427258 pn 0 opn 0) v3 -- ?+0 0x2909a00 -18096> 2013-04-03 10:36:55.890816 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos -18095> 2013-04-03 10:36:55.890833 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 -18094> 2013-04-03 10:36:55.890849 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18093> 2013-04-03 10:36:55.902376 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113222 ==== paxos(lease_ack lc 427300 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (4234496665 0 0) 0x2909a00 con 0x208a840 -18092> 2013-04-03 10:36:55.902418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos -18091> 2013-04-03 10:36:55.902435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 -18090> 2013-04-03 10:36:55.902449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18089> 2013-04-03 10:36:56.926991 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113223 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 52 v187) v2) to leader v1 ==== 560+0+0 (3439395691 0 0) 0x273b8c0 con 0x208a840 -18088> 2013-04-03 10:36:56.927015 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -18087> 2013-04-03 10:36:56.927025 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x273a2c0 from 10.252.24.88:6789/0 -18086> 2013-04-03 10:36:56.927075 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 52 v187) v2 from mds.0 10.252.24.25:6802/3144 -18085> 2013-04-03 10:36:56.927087 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 52 v187) v2 tid 17487) v2 -- ?+0 0x24f8900 con 0x208a840 -18084> 2013-04-03 10:36:57.380716 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 50 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21708 v187) v2 ==== 289+0+0 (4057712754 0 0) 0x29f4dc0 con 0x200d420 -18083> 2013-04-03 10:36:57.380786 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21708 v187) v2 from mds.? 10.252.24.88:6800/17213 -18082> 2013-04-03 10:36:57.380798 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21708 v187) v2 -- ?+0 0x273b8c0 con 0x200d420 -18081> 2013-04-03 10:36:58.333790 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29008 check_down_pgs -18080> 2013-04-03 10:36:58.333819 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29008 v29008: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 667B/s wr, 0op/s -18079> 2013-04-03 10:36:58.333846 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -18078> 2013-04-03 10:36:58.333869 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -18077> 2013-04-03 10:36:58.333905 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -18076> 2013-04-03 10:36:58.333935 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos -18075> 2013-04-03 10:36:58.333953 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 -18074> 2013-04-03 10:36:58.333963 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394012 log -18073> 2013-04-03 10:36:58.333979 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18072> 2013-04-03 10:36:58.334002 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -18071> 2013-04-03 10:36:58.890816 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427300 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2909000 -18070> 2013-04-03 10:36:58.891574 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113224 ==== paxos(lease_ack lc 427300 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (1871708715 0 0) 0x2909000 con 0x208a840 -18069> 2013-04-03 10:36:58.891639 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos -18068> 2013-04-03 10:36:58.891658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 -18067> 2013-04-03 10:36:58.891675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18066> 2013-04-03 10:36:59.581122 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 51 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140338 v187) v2 ==== 289+0+0 (1427439830 0 0) 0x2a52600 con 0x2118420 -18065> 2013-04-03 10:36:59.581195 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140338 v187) v2 from mds.? 10.252.58.31:6800/10908 -18064> 2013-04-03 10:36:59.581208 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140338 v187) v2 -- ?+0 0x29f4dc0 con 0x2118420 -18063> 2013-04-03 10:37:00.927083 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113225 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 53 v187) v2) to leader v1 ==== 560+0+0 (3562205603 0 0) 0x2a528c0 con 0x208a840 -18062> 2013-04-03 10:37:00.927102 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -18061> 2013-04-03 10:37:00.927112 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x273a580 from 10.252.24.88:6789/0 -18060> 2013-04-03 10:37:00.927163 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 53 v187) v2 from mds.0 10.252.24.25:6802/3144 -18059> 2013-04-03 10:37:00.927174 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 53 v187) v2 tid 17488) v2 -- ?+0 0x20adb40 con 0x208a840 -18058> 2013-04-03 10:37:01.380828 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 51 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21709 v187) v2 ==== 289+0+0 (3535107855 0 0) 0x29f4000 con 0x200d420 -18057> 2013-04-03 10:37:01.380898 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21709 v187) v2 from mds.? 10.252.24.88:6800/17213 -18056> 2013-04-03 10:37:01.380910 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21709 v187) v2 -- ?+0 0x2a528c0 con 0x200d420 -18055> 2013-04-03 10:37:01.890885 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427300 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2909780 -18054> 2013-04-03 10:37:01.891638 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113226 ==== paxos(lease_ack lc 427300 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (56673958 0 0) 0x2909780 con 0x208a840 -18053> 2013-04-03 10:37:01.891702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos -18052> 2013-04-03 10:37:01.891721 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 -18051> 2013-04-03 10:37:01.891738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18050> 2013-04-03 10:37:03.334127 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29008 v29008: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 667B/s wr, 0op/s -18049> 2013-04-03 10:37:03.334166 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -18048> 2013-04-03 10:37:03.334189 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -18047> 2013-04-03 10:37:03.334230 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -18046> 2013-04-03 10:37:03.334260 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos -18045> 2013-04-03 10:37:03.334279 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 -18044> 2013-04-03 10:37:03.334286 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394012 log -18043> 2013-04-03 10:37:03.334303 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18042> 2013-04-03 10:37:03.334321 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -18041> 2013-04-03 10:37:03.493840 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113227 ==== forward(pg_stats(1 pgs tid 11 v 602) v1) to leader v1 ==== 757+0+0 (2724222042 0 0) 0x2a52600 con 0x208a840 -18040> 2013-04-03 10:37:03.493862 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 -18039> 2013-04-03 10:37:03.493872 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -18038> 2013-04-03 10:37:03.493924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 preprocess_query pg_stats(1 pgs tid 11 v 602) v1 from osd.2 10.252.24.30:6800/1508 -18037> 2013-04-03 10:37:03.493947 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 prepare_update pg_stats(1 pgs tid 11 v 602) v1 from osd.2 10.252.24.30:6800/1508 -18036> 2013-04-03 10:37:03.493953 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 prepare_pg_stats pg_stats(1 pgs tid 11 v 602) v1 from osd.2 -18035> 2013-04-03 10:37:03.493959 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29008 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -18034> 2013-04-03 10:37:03.544157 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29008 encode_pending v 29009 -18033> 2013-04-03 10:37:03.544199 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427289..427300) queue_proposal bl 654 bytes; ctx = 0x1b91e60 -18032> 2013-04-03 10:37:03.544207 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427289..427300) propose_queued 427301 654 bytes -18031> 2013-04-03 10:37:03.545163 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427300 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc280 -18030> 2013-04-03 10:37:03.578827 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113228 ==== paxos(accept lc 427300 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (601514310 0 0) 0x26dc280 con 0x208a840 -18029> 2013-04-03 10:37:03.580413 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427301 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909780 -18028> 2013-04-03 10:37:03.580446 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427301 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2909000 -18027> 2013-04-03 10:37:03.580517 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29008 update_from_paxos applying incremental 29009 -18026> 2013-04-03 10:37:03.580538 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29009 v29009: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 6300B/s wr, 0op/s -18025> 2013-04-03 10:37:03.580559 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29009 send_pg_creates to 0 pgs -18024> 2013-04-03 10:37:03.580561 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29009 update_logger -18023> 2013-04-03 10:37:03.580571 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29009 create_pending v 29010 -18022> 2013-04-03 10:37:03.580579 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29009 _updated_stats for osd.2 10.252.24.30:6800/1508 -18021> 2013-04-03 10:37:03.580585 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 11) v1 tid 17489) v2 -- ?+0 0x24f8900 con 0x208a840 -18020> 2013-04-03 10:37:03.580598 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29009 check_osd_map already seen 602 >= 602 -18019> 2013-04-03 10:37:03.580600 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29009 update_logger -18018> 2013-04-03 10:37:03.580610 7fcdffcdb700 0 log [INF] : pgmap v29009: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 6300B/s wr, 0op/s -18017> 2013-04-03 10:37:03.580612 7fcdffcdb700 10 send_log to self -18016> 2013-04-03 10:37:03.580613 7fcdffcdb700 10 log_queue is 1 last_log 71 sent 70 num 1 unsent 1 sending 1 -18015> 2013-04-03 10:37:03.580614 7fcdffcdb700 10 will send 2013-04-03 10:37:03.580612 mon.0 10.252.24.25:6789/0 71 : [INF] pgmap v29009: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 6300B/s wr, 0op/s -18014> 2013-04-03 10:37:03.580621 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287ee00 -18013> 2013-04-03 10:37:03.580669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos -18012> 2013-04-03 10:37:03.580688 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 -18011> 2013-04-03 10:37:03.580707 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -18010> 2013-04-03 10:37:03.580729 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287ee00 con 0x1bac160 -18009> 2013-04-03 10:37:03.580751 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos -18008> 2013-04-03 10:37:03.580765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 -18007> 2013-04-03 10:37:03.580774 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18006> 2013-04-03 10:37:03.580789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 preprocess_log log(1 entries) v1 from mon.0 -18005> 2013-04-03 10:37:03.580810 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -18004> 2013-04-03 10:37:03.580822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 prepare_log log(1 entries) v1 from mon.0 -18003> 2013-04-03 10:37:03.580831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 logging 2013-04-03 10:37:03.580612 mon.0 10.252.24.25:6789/0 71 : [INF] pgmap v29009: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 6300B/s wr, 0op/s -18002> 2013-04-03 10:37:03.581199 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 52 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140339 v187) v2 ==== 289+0+0 (1987821483 0 0) 0x2a52340 con 0x2118420 -18001> 2013-04-03 10:37:03.581256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140339 v187) v2 from mds.? 10.252.58.31:6800/10908 -18000> 2013-04-03 10:37:03.581266 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140339 v187) v2 -- ?+0 0x2a52600 con 0x2118420 -17999> 2013-04-03 10:37:03.631158 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113229 ==== paxos(lease_ack lc 427301 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (1926083364 0 0) 0x2909000 con 0x208a840 -17998> 2013-04-03 10:37:03.631205 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos -17997> 2013-04-03 10:37:03.631223 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394012 update_from_paxos version 394012 summary v 394012 -17996> 2013-04-03 10:37:03.631238 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -17995> 2013-04-03 10:37:04.579027 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394012 encode_pending v394013 -17994> 2013-04-03 10:37:04.579055 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427289..427301) queue_proposal bl 443 bytes; ctx = 0x1b91e70 -17993> 2013-04-03 10:37:04.579062 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427289..427301) propose_queued 427302 443 bytes -17992> 2013-04-03 10:37:04.581027 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427301 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dcc80 -17991> 2013-04-03 10:37:04.609441 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113230 ==== paxos(accept lc 427301 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3341247151 0 0) 0x26dcc80 con 0x208a840 -17990> 2013-04-03 10:37:04.609769 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427302 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909000 -17989> 2013-04-03 10:37:04.609802 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427302 fc 427289 pn 0 opn 0) v3 -- ?+0 0x26dc280 -17988> 2013-04-03 10:37:04.609843 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos -17987> 2013-04-03 10:37:04.609870 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos version 394013 summary v 394012 -17986> 2013-04-03 10:37:04.609884 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos latest full 393798 -17985> 2013-04-03 10:37:04.609900 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394013 update_from_paxos applying incremental log 394013 2013-04-03 10:37:03.580612 mon.0 10.252.24.25:6789/0 71 : [INF] pgmap v29009: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 6300B/s wr, 0op/s -17984> 2013-04-03 10:37:04.609962 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 check_subs -17983> 2013-04-03 10:37:04.609995 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 create_pending v 394014 -17982> 2013-04-03 10:37:04.610017 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394013 _updated_log for mon.0 10.252.24.25:6789/0 -17981> 2013-04-03 10:37:04.610030 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 71) v1 -- ?+0 0x28a51c0 con 0x1bac160 -17980> 2013-04-03 10:37:04.610080 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos -17979> 2013-04-03 10:37:04.610096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos version 394013 summary v 394013 -17978> 2013-04-03 10:37:04.610112 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -17977> 2013-04-03 10:37:04.610134 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 71) v1 ==== 0+0+0 (0 0 0) 0x28a51c0 con 0x1bac160 -17976> 2013-04-03 10:37:04.610147 7fcdffcdb700 10 handle_log_ack log(last 71) v1 -17975> 2013-04-03 10:37:04.610150 7fcdffcdb700 10 logged 2013-04-03 10:37:03.580612 mon.0 10.252.24.25:6789/0 71 : [INF] pgmap v29009: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 6300B/s wr, 0op/s -17974> 2013-04-03 10:37:04.621792 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113231 ==== paxos(lease_ack lc 427302 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (652358494 0 0) 0x26dc280 con 0x208a840 -17973> 2013-04-03 10:37:04.621834 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos -17972> 2013-04-03 10:37:04.621850 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos version 394013 summary v 394013 -17971> 2013-04-03 10:37:04.621865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -17970> 2013-04-03 10:37:04.927203 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113232 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 54 v187) v2) to leader v1 ==== 560+0+0 (2208094868 0 0) 0x273adc0 con 0x208a840 -17969> 2013-04-03 10:37:04.927219 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -17968> 2013-04-03 10:37:04.927227 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2996b80 from 10.252.24.88:6789/0 -17967> 2013-04-03 10:37:04.927259 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 54 v187) v2 from mds.0 10.252.24.25:6802/3144 -17966> 2013-04-03 10:37:04.927269 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 54 v187) v2 tid 17490) v2 -- ?+0 0x20adb40 con 0x208a840 -17965> 2013-04-03 10:37:05.380925 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 52 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21710 v187) v2 ==== 289+0+0 (3070670472 0 0) 0x29f4580 con 0x200d420 -17964> 2013-04-03 10:37:05.381003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21710 v187) v2 from mds.? 10.252.24.88:6800/17213 -17963> 2013-04-03 10:37:05.381016 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21710 v187) v2 -- ?+0 0x273adc0 con 0x200d420 -17962> 2013-04-03 10:37:07.581377 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 53 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140340 v187) v2 ==== 289+0+0 (2658304984 0 0) 0x2a52080 con 0x2118420 -17961> 2013-04-03 10:37:07.581451 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140340 v187) v2 from mds.? 10.252.58.31:6800/10908 -17960> 2013-04-03 10:37:07.581463 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140340 v187) v2 -- ?+0 0x29f4580 con 0x2118420 -17959> 2013-04-03 10:37:07.609874 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427302 fc 427289 pn 0 opn 0) v3 -- ?+0 0x20f4c80 -17958> 2013-04-03 10:37:07.610615 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113233 ==== paxos(lease_ack lc 427302 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (2094209312 0 0) 0x20f4c80 con 0x208a840 -17957> 2013-04-03 10:37:07.610675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos -17956> 2013-04-03 10:37:07.610695 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos version 394013 summary v 394013 -17955> 2013-04-03 10:37:07.610712 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -17954> 2013-04-03 10:37:08.334466 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29009 check_down_pgs -17953> 2013-04-03 10:37:08.334496 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29009 v29009: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 6300B/s wr, 0op/s -17952> 2013-04-03 10:37:08.334523 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -17951> 2013-04-03 10:37:08.334545 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 e602: 5 osds: 5 up, 5 in -17950> 2013-04-03 10:37:08.334581 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e602 min_last_epoch_clean 593 -17949> 2013-04-03 10:37:08.334612 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos -17948> 2013-04-03 10:37:08.334630 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394013 update_from_paxos version 394013 summary v 394013 -17947> 2013-04-03 10:37:08.334639 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394013 log -17946> 2013-04-03 10:37:08.334656 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -17945> 2013-04-03 10:37:08.334674 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -17944> 2013-04-03 10:37:08.927357 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113234 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 55 v187) v2) to leader v1 ==== 560+0+0 (2329216151 0 0) 0x2a52340 con 0x208a840 -17943> 2013-04-03 10:37:08.927379 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -17942> 2013-04-03 10:37:08.927388 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2995000 from 10.252.24.88:6789/0 -17941> 2013-04-03 10:37:08.927439 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 55 v187) v2 from mds.0 10.252.24.25:6802/3144 -17940> 2013-04-03 10:37:08.927451 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 55 v187) v2 tid 17491) v2 -- ?+0 0x24f8900 con 0x208a840 -17939> 2013-04-03 10:37:09.193104 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113235 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (2584856398 0 0) 0x2a52080 con 0x208a840 -17938> 2013-04-03 10:37:09.193131 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -17937> 2013-04-03 10:37:09.193138 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -17936> 2013-04-03 10:37:09.193147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 -17935> 2013-04-03 10:37:09.193152 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( start_reply flags( reply-to(mon.2 10.252.24.88:6789/0) ) ) v1 -- ?+0 0x2a52340 -17934> 2013-04-03 10:37:09.381027 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 53 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21711 v187) v2 ==== 289+0+0 (2489846261 0 0) 0x29f4b00 con 0x200d420 -17933> 2013-04-03 10:37:09.381095 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21711 v187) v2 from mds.? 10.252.24.88:6800/17213 -17932> 2013-04-03 10:37:09.381108 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21711 v187) v2 -- ?+0 0x2a52080 con 0x200d420 -17931> 2013-04-03 10:37:09.572986 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113236 ==== forward(pg_stats(2 pgs tid 19 v 602) v1) to leader v1 ==== 1130+0+0 (3859428727 0 0) 0x2a84b80 con 0x208a840 -17930> 2013-04-03 10:37:09.573007 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -17929> 2013-04-03 10:37:09.573018 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -17928> 2013-04-03 10:37:09.573067 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29009 preprocess_query pg_stats(2 pgs tid 19 v 602) v1 from osd.1 10.252.24.25:6805/3572 -17927> 2013-04-03 10:37:09.573091 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29009 prepare_update pg_stats(2 pgs tid 19 v 602) v1 from osd.1 10.252.24.25:6805/3572 -17926> 2013-04-03 10:37:09.573097 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29009 prepare_pg_stats pg_stats(2 pgs tid 19 v 602) v1 from osd.1 -17925> 2013-04-03 10:37:09.573104 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29009 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -17924> 2013-04-03 10:37:09.623308 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29009 encode_pending v 29010 -17923> 2013-04-03 10:37:09.623357 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427302) queue_proposal bl 1027 bytes; ctx = 0x1b91d90 -17922> 2013-04-03 10:37:09.623366 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427302) propose_queued 427303 1027 bytes -17921> 2013-04-03 10:37:09.658552 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427302 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787280 -17920> 2013-04-03 10:37:09.690004 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113237 ==== paxos(accept lc 427302 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3971456464 0 0) 0x2787280 con 0x208a840 -17919> 2013-04-03 10:37:09.722195 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427303 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4c80 -17918> 2013-04-03 10:37:09.722224 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427303 fc 427289 pn 0 opn 0) v3 -- ?+0 0x26dc280 -17917> 2013-04-03 10:37:09.722307 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29009 update_from_paxos applying incremental 29010 -17916> 2013-04-03 10:37:09.722330 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 v29010: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 5460B/s wr, 0op/s -17915> 2013-04-03 10:37:09.722350 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 send_pg_creates to 0 pgs -17914> 2013-04-03 10:37:09.722354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 update_logger -17913> 2013-04-03 10:37:09.722363 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 create_pending v 29011 -17912> 2013-04-03 10:37:09.722373 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 _updated_stats for osd.1 10.252.24.25:6805/3572 -17911> 2013-04-03 10:37:09.722382 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(2 pgs tid 19) v1 tid 17492) v2 -- ?+0 0x20adb40 con 0x208a840 -17910> 2013-04-03 10:37:09.722392 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 check_osd_map already seen 602 >= 602 -17909> 2013-04-03 10:37:09.722396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 update_logger -17908> 2013-04-03 10:37:09.722407 7fcdffcdb700 0 log [INF] : pgmap v29010: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 5460B/s wr, 0op/s -17907> 2013-04-03 10:37:09.722408 7fcdffcdb700 10 send_log to self -17906> 2013-04-03 10:37:09.722409 7fcdffcdb700 10 log_queue is 1 last_log 72 sent 71 num 1 unsent 1 sending 1 -17905> 2013-04-03 10:37:09.722411 7fcdffcdb700 10 will send 2013-04-03 10:37:09.722408 mon.0 10.252.24.25:6789/0 72 : [INF] pgmap v29010: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 5460B/s wr, 0op/s -17904> 2013-04-03 10:37:09.722417 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287ee00 -17903> 2013-04-03 10:37:09.722466 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 update_from_paxos -17902> 2013-04-03 10:37:09.722486 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 update_from_paxos version 394013 summary v 394013 -17901> 2013-04-03 10:37:09.722505 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17900> 2013-04-03 10:37:09.722527 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287ee00 con 0x1bac160 -17899> 2013-04-03 10:37:09.722549 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 update_from_paxos -17898> 2013-04-03 10:37:09.722564 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 update_from_paxos version 394013 summary v 394013 -17897> 2013-04-03 10:37:09.722573 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -17896> 2013-04-03 10:37:09.722584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 preprocess_log log(1 entries) v1 from mon.0 -17895> 2013-04-03 10:37:09.722606 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -17894> 2013-04-03 10:37:09.722618 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 prepare_log log(1 entries) v1 from mon.0 -17893> 2013-04-03 10:37:09.722628 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 logging 2013-04-03 10:37:09.722408 mon.0 10.252.24.25:6789/0 72 : [INF] pgmap v29010: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 5460B/s wr, 0op/s -17892> 2013-04-03 10:37:09.749678 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113238 ==== paxos(lease_ack lc 427303 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (270929864 0 0) 0x26dc280 con 0x208a840 -17891> 2013-04-03 10:37:09.749719 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 update_from_paxos -17890> 2013-04-03 10:37:09.749736 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 update_from_paxos version 394013 summary v 394013 -17889> 2013-04-03 10:37:09.749751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17888> 2013-04-03 10:37:09.750031 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113239 ==== forward(pg_stats(1 pgs tid 30440 v 602) v1) to leader v1 ==== 757+0+0 (3571705875 0 0) 0x2a848c0 con 0x208a840 -17887> 2013-04-03 10:37:09.750042 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -17886> 2013-04-03 10:37:09.750050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20adb40 from 10.252.24.88:6789/0 -17885> 2013-04-03 10:37:09.750076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 preprocess_query pg_stats(1 pgs tid 30440 v 602) v1 from osd.3 10.252.24.88:6801/17538 -17884> 2013-04-03 10:37:09.750097 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 prepare_update pg_stats(1 pgs tid 30440 v 602) v1 from osd.3 10.252.24.88:6801/17538 -17883> 2013-04-03 10:37:09.750103 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 prepare_pg_stats pg_stats(1 pgs tid 30440 v 602) v1 from osd.3 -17882> 2013-04-03 10:37:09.750108 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -17881> 2013-04-03 10:37:10.361212 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x29b4000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:32989/0 -17880> 2013-04-03 10:37:10.361243 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:32989/0) -17879> 2013-04-03 10:37:10.361257 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4000 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -17878> 2013-04-03 10:37:10.361468 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4000 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -17877> 2013-04-03 10:37:10.690207 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394013 encode_pending v394014 -17876> 2013-04-03 10:37:10.690246 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427303) queue_proposal bl 443 bytes; ctx = 0x1b91c10 -17875> 2013-04-03 10:37:10.690255 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427303) propose_queued 427304 443 bytes -17874> 2013-04-03 10:37:10.691363 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427303 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787500 -17873> 2013-04-03 10:37:10.691462 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 encode_pending v 29011 -17872> 2013-04-03 10:37:10.691497 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427289..427303) queue_proposal bl 654 bytes; ctx = 0x1b91e40 -17871> 2013-04-03 10:37:10.691504 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427289..427303) propose_new_value not active; proposal queued -17870> 2013-04-03 10:37:10.741172 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113240 ==== paxos(accept lc 427303 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (644060495 0 0) 0x2787500 con 0x208a840 -17869> 2013-04-03 10:37:10.741505 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427304 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc280 -17868> 2013-04-03 10:37:10.741540 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427304 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2787280 -17867> 2013-04-03 10:37:10.741579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos -17866> 2013-04-03 10:37:10.741606 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos version 394014 summary v 394013 -17865> 2013-04-03 10:37:10.741624 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos latest full 393798 -17864> 2013-04-03 10:37:10.741644 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos applying incremental log 394014 2013-04-03 10:37:09.722408 mon.0 10.252.24.25:6789/0 72 : [INF] pgmap v29010: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 5460B/s wr, 0op/s -17863> 2013-04-03 10:37:10.741694 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 check_subs -17862> 2013-04-03 10:37:10.741726 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 create_pending v 394015 -17861> 2013-04-03 10:37:10.741749 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 _updated_log for mon.0 10.252.24.25:6789/0 -17860> 2013-04-03 10:37:10.741763 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 72) v1 -- ?+0 0x28a5540 con 0x1bac160 -17859> 2013-04-03 10:37:10.741787 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427304) propose_queued 427305 654 bytes -17858> 2013-04-03 10:37:10.742526 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427304 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dcc80 -17857> 2013-04-03 10:37:10.742560 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 72) v1 ==== 0+0+0 (0 0 0) 0x28a5540 con 0x1bac160 -17856> 2013-04-03 10:37:10.742577 7fcdffcdb700 10 handle_log_ack log(last 72) v1 -17855> 2013-04-03 10:37:10.742579 7fcdffcdb700 10 logged 2013-04-03 10:37:09.722408 mon.0 10.252.24.25:6789/0 72 : [INF] pgmap v29010: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 5460B/s wr, 0op/s -17854> 2013-04-03 10:37:10.753494 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113241 ==== paxos(lease_ack lc 427304 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (2108229741 0 0) 0x26dcc80 con 0x208a840 -17853> 2013-04-03 10:37:10.767166 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113242 ==== paxos(accept lc 427304 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3097634712 0 0) 0x2787280 con 0x208a840 -17852> 2013-04-03 10:37:10.768400 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427305 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dcc80 -17851> 2013-04-03 10:37:10.768435 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427305 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2787500 -17850> 2013-04-03 10:37:10.768503 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29010 update_from_paxos applying incremental 29011 -17849> 2013-04-03 10:37:10.768522 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 v29011: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 716B/s wr, 0op/s -17848> 2013-04-03 10:37:10.768542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 send_pg_creates to 0 pgs -17847> 2013-04-03 10:37:10.768545 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 update_logger -17846> 2013-04-03 10:37:10.768554 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 create_pending v 29012 -17845> 2013-04-03 10:37:10.768563 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 _updated_stats for osd.3 10.252.24.88:6801/17538 -17844> 2013-04-03 10:37:10.768569 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30440) v1 tid 17493) v2 -- ?+0 0x24f8900 con 0x208a840 -17843> 2013-04-03 10:37:10.768581 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 check_osd_map already seen 602 >= 602 -17842> 2013-04-03 10:37:10.768584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 update_logger -17841> 2013-04-03 10:37:10.768594 7fcdffcdb700 0 log [INF] : pgmap v29011: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 716B/s wr, 0op/s -17840> 2013-04-03 10:37:10.768596 7fcdffcdb700 10 send_log to self -17839> 2013-04-03 10:37:10.768596 7fcdffcdb700 10 log_queue is 1 last_log 73 sent 72 num 1 unsent 1 sending 1 -17838> 2013-04-03 10:37:10.768598 7fcdffcdb700 10 will send 2013-04-03 10:37:10.768595 mon.0 10.252.24.25:6789/0 73 : [INF] pgmap v29011: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 716B/s wr, 0op/s -17837> 2013-04-03 10:37:10.768604 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287e400 -17836> 2013-04-03 10:37:10.768650 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos -17835> 2013-04-03 10:37:10.768668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos version 394014 summary v 394014 -17834> 2013-04-03 10:37:10.768687 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17833> 2013-04-03 10:37:10.768707 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287e400 con 0x1bac160 -17832> 2013-04-03 10:37:10.768729 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos -17831> 2013-04-03 10:37:10.768744 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos version 394014 summary v 394014 -17830> 2013-04-03 10:37:10.768753 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -17829> 2013-04-03 10:37:10.768764 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 preprocess_log log(1 entries) v1 from mon.0 -17828> 2013-04-03 10:37:10.768782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -17827> 2013-04-03 10:37:10.768793 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 prepare_log log(1 entries) v1 from mon.0 -17826> 2013-04-03 10:37:10.768802 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 logging 2013-04-03 10:37:10.768595 mon.0 10.252.24.25:6789/0 73 : [INF] pgmap v29011: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 716B/s wr, 0op/s -17825> 2013-04-03 10:37:10.793300 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113243 ==== paxos(lease_ack lc 427305 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (2020296042 0 0) 0x2787500 con 0x208a840 -17824> 2013-04-03 10:37:10.793348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos -17823> 2013-04-03 10:37:10.793366 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 update_from_paxos version 394014 summary v 394014 -17822> 2013-04-03 10:37:10.793381 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17821> 2013-04-03 10:37:11.581434 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 54 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140341 v187) v2 ==== 289+0+0 (3172744357 0 0) 0x2a51dc0 con 0x2118420 -17820> 2013-04-03 10:37:11.581504 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140341 v187) v2 from mds.? 10.252.58.31:6800/10908 -17819> 2013-04-03 10:37:11.581518 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140341 v187) v2 -- ?+0 0x2a848c0 con 0x2118420 -17818> 2013-04-03 10:37:11.767366 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394014 encode_pending v394015 -17817> 2013-04-03 10:37:11.767395 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427305) queue_proposal bl 442 bytes; ctx = 0x1b91e50 -17816> 2013-04-03 10:37:11.767403 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427305) propose_queued 427306 442 bytes -17815> 2013-04-03 10:37:11.768518 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427305 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787780 -17814> 2013-04-03 10:37:11.801707 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113244 ==== paxos(accept lc 427305 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (985192712 0 0) 0x2787780 con 0x208a840 -17813> 2013-04-03 10:37:11.802026 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427306 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787500 -17812> 2013-04-03 10:37:11.802057 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427306 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2787280 -17811> 2013-04-03 10:37:11.802096 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos -17810> 2013-04-03 10:37:11.802124 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394014 -17809> 2013-04-03 10:37:11.802139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos latest full 393798 -17808> 2013-04-03 10:37:11.802156 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos applying incremental log 394015 2013-04-03 10:37:10.768595 mon.0 10.252.24.25:6789/0 73 : [INF] pgmap v29011: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 716B/s wr, 0op/s -17807> 2013-04-03 10:37:11.802204 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 check_subs -17806> 2013-04-03 10:37:11.802235 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 create_pending v 394016 -17805> 2013-04-03 10:37:11.802257 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 _updated_log for mon.0 10.252.24.25:6789/0 -17804> 2013-04-03 10:37:11.802270 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 73) v1 -- ?+0 0x28a5540 con 0x1bac160 -17803> 2013-04-03 10:37:11.802320 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos -17802> 2013-04-03 10:37:11.802336 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 -17801> 2013-04-03 10:37:11.802357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17800> 2013-04-03 10:37:11.802379 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 73) v1 ==== 0+0+0 (0 0 0) 0x28a5540 con 0x1bac160 -17799> 2013-04-03 10:37:11.802387 7fcdffcdb700 10 handle_log_ack log(last 73) v1 -17798> 2013-04-03 10:37:11.802388 7fcdffcdb700 10 logged 2013-04-03 10:37:10.768595 mon.0 10.252.24.25:6789/0 73 : [INF] pgmap v29011: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 716B/s wr, 0op/s -17797> 2013-04-03 10:37:11.814037 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113245 ==== paxos(lease_ack lc 427306 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (3178233102 0 0) 0x2787280 con 0x208a840 -17796> 2013-04-03 10:37:11.814072 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos -17795> 2013-04-03 10:37:11.814088 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 -17794> 2013-04-03 10:37:11.814103 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17793> 2013-04-03 10:37:12.927485 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113246 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 56 v187) v2) to leader v1 ==== 560+0+0 (221567120 0 0) 0x2a84600 con 0x208a840 -17792> 2013-04-03 10:37:12.927507 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -17791> 2013-04-03 10:37:12.927517 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a84340 from 10.252.24.88:6789/0 -17790> 2013-04-03 10:37:12.927569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 56 v187) v2 from mds.0 10.252.24.25:6802/3144 -17789> 2013-04-03 10:37:12.927581 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 56 v187) v2 tid 17494) v2 -- ?+0 0x20adb40 con 0x208a840 -17788> 2013-04-03 10:37:13.334817 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 check_down_pgs -17787> 2013-04-03 10:37:13.334851 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 v29011: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 716B/s wr, 0op/s -17786> 2013-04-03 10:37:13.334879 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -17785> 2013-04-03 10:37:13.334902 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in -17784> 2013-04-03 10:37:13.334940 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -17783> 2013-04-03 10:37:13.334971 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos -17782> 2013-04-03 10:37:13.334990 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 -17781> 2013-04-03 10:37:13.335004 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 log -17780> 2013-04-03 10:37:13.335021 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17779> 2013-04-03 10:37:13.335040 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -17778> 2013-04-03 10:37:13.381114 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 54 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21712 v187) v2 ==== 289+0+0 (1607180973 0 0) 0x29f58c0 con 0x200d420 -17777> 2013-04-03 10:37:13.381165 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21712 v187) v2 from mds.? 10.252.24.88:6800/17213 -17776> 2013-04-03 10:37:13.381177 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21712 v187) v2 -- ?+0 0x29c58c0 con 0x200d420 -17775> 2013-04-03 10:37:14.802134 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427306 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2787a00 -17774> 2013-04-03 10:37:14.803053 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113247 ==== paxos(lease_ack lc 427306 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (2335220110 0 0) 0x2787a00 con 0x208a840 -17773> 2013-04-03 10:37:14.803118 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos -17772> 2013-04-03 10:37:14.803139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 -17771> 2013-04-03 10:37:14.803159 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17770> 2013-04-03 10:37:15.581573 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 55 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140342 v187) v2 ==== 289+0+0 (3635412258 0 0) 0x2a51b00 con 0x2118420 -17769> 2013-04-03 10:37:15.581644 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140342 v187) v2 from mds.? 10.252.58.31:6800/10908 -17768> 2013-04-03 10:37:15.581658 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140342 v187) v2 -- ?+0 0x29f58c0 con 0x2118420 -17767> 2013-04-03 10:37:16.927616 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113248 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 57 v187) v2) to leader v1 ==== 560+0+0 (75302547 0 0) 0x2a51dc0 con 0x208a840 -17766> 2013-04-03 10:37:16.927635 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -17765> 2013-04-03 10:37:16.927646 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a84080 from 10.252.24.88:6789/0 -17764> 2013-04-03 10:37:16.927698 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 57 v187) v2 from mds.0 10.252.24.25:6802/3144 -17763> 2013-04-03 10:37:16.927709 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 57 v187) v2 tid 17495) v2 -- ?+0 0x24f8900 con 0x208a840 -17762> 2013-04-03 10:37:17.381257 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 55 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21713 v187) v2 ==== 289+0+0 (2091195856 0 0) 0x29c4dc0 con 0x200d420 -17761> 2013-04-03 10:37:17.381537 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21713 v187) v2 from mds.? 10.252.24.88:6800/17213 -17760> 2013-04-03 10:37:17.381555 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21713 v187) v2 -- ?+0 0x2a51dc0 con 0x200d420 -17759> 2013-04-03 10:37:17.802206 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427306 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2787c80 -17758> 2013-04-03 10:37:17.803049 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113249 ==== paxos(lease_ack lc 427306 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (3596365997 0 0) 0x2787c80 con 0x208a840 -17757> 2013-04-03 10:37:17.803117 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos -17756> 2013-04-03 10:37:17.803137 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 -17755> 2013-04-03 10:37:17.803155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17754> 2013-04-03 10:37:18.335183 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 v29011: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 716B/s wr, 0op/s -17753> 2013-04-03 10:37:18.335229 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -17752> 2013-04-03 10:37:18.335253 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in -17751> 2013-04-03 10:37:18.335296 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -17750> 2013-04-03 10:37:18.335328 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos -17749> 2013-04-03 10:37:18.335347 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 -17748> 2013-04-03 10:37:18.335355 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 log -17747> 2013-04-03 10:37:18.335372 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17746> 2013-04-03 10:37:18.335389 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -17745> 2013-04-03 10:37:18.495870 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113250 ==== forward(pg_stats(1 pgs tid 12 v 602) v1) to leader v1 ==== 757+0+0 (1339368 0 0) 0x2a51b00 con 0x208a840 -17744> 2013-04-03 10:37:18.495886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6800/1508 via mon.2 10.252.24.88:6789/0 -17743> 2013-04-03 10:37:18.495895 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -17742> 2013-04-03 10:37:18.495931 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 preprocess_query pg_stats(1 pgs tid 12 v 602) v1 from osd.2 10.252.24.30:6800/1508 -17741> 2013-04-03 10:37:18.495953 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 prepare_update pg_stats(1 pgs tid 12 v 602) v1 from osd.2 10.252.24.30:6800/1508 -17740> 2013-04-03 10:37:18.495959 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 prepare_pg_stats pg_stats(1 pgs tid 12 v 602) v1 from osd.2 -17739> 2013-04-03 10:37:18.495966 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -17738> 2013-04-03 10:37:18.546146 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 encode_pending v 29012 -17737> 2013-04-03 10:37:18.546184 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427306) queue_proposal bl 654 bytes; ctx = 0x1b91e30 -17736> 2013-04-03 10:37:18.546194 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427306) propose_queued 427307 654 bytes -17735> 2013-04-03 10:37:18.547289 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427306 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4280 -17734> 2013-04-03 10:37:18.579547 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113251 ==== paxos(accept lc 427306 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3321092350 0 0) 0x29b4280 con 0x208a840 -17733> 2013-04-03 10:37:18.579874 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427307 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787c80 -17732> 2013-04-03 10:37:18.579902 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427307 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2787a00 -17731> 2013-04-03 10:37:18.579975 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29011 update_from_paxos applying incremental 29012 -17730> 2013-04-03 10:37:18.579992 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 v29012: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3328B/s wr, 0op/s -17729> 2013-04-03 10:37:18.580009 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 send_pg_creates to 0 pgs -17728> 2013-04-03 10:37:18.580012 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 update_logger -17727> 2013-04-03 10:37:18.580022 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 create_pending v 29013 -17726> 2013-04-03 10:37:18.580031 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 _updated_stats for osd.2 10.252.24.30:6800/1508 -17725> 2013-04-03 10:37:18.580037 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 12) v1 tid 17496) v2 -- ?+0 0x20adb40 con 0x208a840 -17724> 2013-04-03 10:37:18.580047 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 check_osd_map already seen 602 >= 602 -17723> 2013-04-03 10:37:18.580049 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 update_logger -17722> 2013-04-03 10:37:18.580060 7fcdffcdb700 0 log [INF] : pgmap v29012: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3328B/s wr, 0op/s -17721> 2013-04-03 10:37:18.580061 7fcdffcdb700 10 send_log to self -17720> 2013-04-03 10:37:18.580062 7fcdffcdb700 10 log_queue is 1 last_log 74 sent 73 num 1 unsent 1 sending 1 -17719> 2013-04-03 10:37:18.580069 7fcdffcdb700 10 will send 2013-04-03 10:37:18.580061 mon.0 10.252.24.25:6789/0 74 : [INF] pgmap v29012: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3328B/s wr, 0op/s -17718> 2013-04-03 10:37:18.580076 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287e400 -17717> 2013-04-03 10:37:18.580121 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos -17716> 2013-04-03 10:37:18.580139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 -17715> 2013-04-03 10:37:18.580155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17714> 2013-04-03 10:37:18.580178 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287e400 con 0x1bac160 -17713> 2013-04-03 10:37:18.580200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos -17712> 2013-04-03 10:37:18.580215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 -17711> 2013-04-03 10:37:18.580224 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -17710> 2013-04-03 10:37:18.580236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 preprocess_log log(1 entries) v1 from mon.0 -17709> 2013-04-03 10:37:18.580258 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -17708> 2013-04-03 10:37:18.580269 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 prepare_log log(1 entries) v1 from mon.0 -17707> 2013-04-03 10:37:18.580279 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 logging 2013-04-03 10:37:18.580061 mon.0 10.252.24.25:6789/0 74 : [INF] pgmap v29012: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3328B/s wr, 0op/s -17706> 2013-04-03 10:37:18.605628 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113252 ==== paxos(lease_ack lc 427307 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (491868676 0 0) 0x2787a00 con 0x208a840 -17705> 2013-04-03 10:37:18.605664 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos -17704> 2013-04-03 10:37:18.605681 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 update_from_paxos version 394015 summary v 394015 -17703> 2013-04-03 10:37:18.605696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17702> 2013-04-03 10:37:19.579748 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394015 encode_pending v394016 -17701> 2013-04-03 10:37:19.579779 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427307) queue_proposal bl 443 bytes; ctx = 0x1b91d80 -17700> 2013-04-03 10:37:19.579787 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427307) propose_queued 427308 443 bytes -17699> 2013-04-03 10:37:19.580901 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427307 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4000 -17698> 2013-04-03 10:37:19.581715 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 56 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140343 v187) v2 ==== 289+0+0 (4223746655 0 0) 0x2a51840 con 0x2118420 -17697> 2013-04-03 10:37:19.581769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140343 v187) v2 from mds.? 10.252.58.31:6800/10908 -17696> 2013-04-03 10:37:19.581784 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140343 v187) v2 -- ?+0 0x2a51b00 con 0x2118420 -17695> 2013-04-03 10:37:19.605677 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113253 ==== paxos(accept lc 427307 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (846334408 0 0) 0x29b4000 con 0x208a840 -17694> 2013-04-03 10:37:19.606023 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427308 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787a00 -17693> 2013-04-03 10:37:19.606057 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427308 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29b4280 -17692> 2013-04-03 10:37:19.606091 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos -17691> 2013-04-03 10:37:19.606118 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394015 -17690> 2013-04-03 10:37:19.606133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos latest full 393798 -17689> 2013-04-03 10:37:19.606152 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos applying incremental log 394016 2013-04-03 10:37:18.580061 mon.0 10.252.24.25:6789/0 74 : [INF] pgmap v29012: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3328B/s wr, 0op/s -17688> 2013-04-03 10:37:19.606200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 check_subs -17687> 2013-04-03 10:37:19.606231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 create_pending v 394017 -17686> 2013-04-03 10:37:19.606254 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 _updated_log for mon.0 10.252.24.25:6789/0 -17685> 2013-04-03 10:37:19.606267 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 74) v1 -- ?+0 0x28a5700 con 0x1bac160 -17684> 2013-04-03 10:37:19.606315 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos -17683> 2013-04-03 10:37:19.606331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394016 -17682> 2013-04-03 10:37:19.606349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17681> 2013-04-03 10:37:19.606371 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 74) v1 ==== 0+0+0 (0 0 0) 0x28a5700 con 0x1bac160 -17680> 2013-04-03 10:37:19.606379 7fcdffcdb700 10 handle_log_ack log(last 74) v1 -17679> 2013-04-03 10:37:19.606381 7fcdffcdb700 10 logged 2013-04-03 10:37:18.580061 mon.0 10.252.24.25:6789/0 74 : [INF] pgmap v29012: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3328B/s wr, 0op/s -17678> 2013-04-03 10:37:19.619342 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113254 ==== paxos(lease_ack lc 427308 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (972142269 0 0) 0x29b4280 con 0x208a840 -17677> 2013-04-03 10:37:19.619386 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos -17676> 2013-04-03 10:37:19.619403 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394016 -17675> 2013-04-03 10:37:19.619418 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17674> 2013-04-03 10:37:20.927711 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113255 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 58 v187) v2) to leader v1 ==== 560+0+0 (225762287 0 0) 0x2a83dc0 con 0x208a840 -17673> 2013-04-03 10:37:20.927733 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -17672> 2013-04-03 10:37:20.927744 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83b00 from 10.252.24.88:6789/0 -17671> 2013-04-03 10:37:20.927795 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 58 v187) v2 from mds.0 10.252.24.25:6802/3144 -17670> 2013-04-03 10:37:20.927807 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 58 v187) v2 tid 17497) v2 -- ?+0 0x24f8900 con 0x208a840 -17669> 2013-04-03 10:37:21.381366 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 56 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21714 v187) v2 ==== 289+0+0 (420928599 0 0) 0x29c4000 con 0x200d420 -17668> 2013-04-03 10:37:21.381435 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21714 v187) v2 from mds.? 10.252.24.88:6800/17213 -17667> 2013-04-03 10:37:21.381448 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21714 v187) v2 -- ?+0 0x2a83dc0 con 0x200d420 -17666> 2013-04-03 10:37:22.606136 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427308 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2ab4c80 -17665> 2013-04-03 10:37:22.607359 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113256 ==== paxos(lease_ack lc 427308 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (2580981994 0 0) 0x2ab4c80 con 0x208a840 -17664> 2013-04-03 10:37:22.607431 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos -17663> 2013-04-03 10:37:22.607452 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394016 -17662> 2013-04-03 10:37:22.607472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17661> 2013-04-03 10:37:23.335536 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 check_down_pgs -17660> 2013-04-03 10:37:23.335564 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 v29012: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 3328B/s wr, 0op/s -17659> 2013-04-03 10:37:23.335592 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -17658> 2013-04-03 10:37:23.335617 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in -17657> 2013-04-03 10:37:23.335655 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -17656> 2013-04-03 10:37:23.335686 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos -17655> 2013-04-03 10:37:23.335705 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394016 -17654> 2013-04-03 10:37:23.335715 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 log -17653> 2013-04-03 10:37:23.335732 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17652> 2013-04-03 10:37:23.335751 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -17651> 2013-04-03 10:37:23.581829 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 57 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140344 v187) v2 ==== 289+0+0 (223474485 0 0) 0x2a51580 con 0x2118420 -17650> 2013-04-03 10:37:23.581875 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140344 v187) v2 from mds.? 10.252.58.31:6800/10908 -17649> 2013-04-03 10:37:23.581886 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140344 v187) v2 -- ?+0 0x29c4000 con 0x2118420 -17648> 2013-04-03 10:37:24.574340 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113257 ==== forward(pg_stats(1 pgs tid 20 v 602) v1) to leader v1 ==== 757+0+0 (2864642805 0 0) 0x2a51840 con 0x208a840 -17647> 2013-04-03 10:37:24.574364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -17646> 2013-04-03 10:37:24.574374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -17645> 2013-04-03 10:37:24.574426 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 preprocess_query pg_stats(1 pgs tid 20 v 602) v1 from osd.1 10.252.24.25:6805/3572 -17644> 2013-04-03 10:37:24.574450 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 prepare_update pg_stats(1 pgs tid 20 v 602) v1 from osd.1 10.252.24.25:6805/3572 -17643> 2013-04-03 10:37:24.574456 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 prepare_pg_stats pg_stats(1 pgs tid 20 v 602) v1 from osd.1 -17642> 2013-04-03 10:37:24.574463 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -17641> 2013-04-03 10:37:24.624671 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 encode_pending v 29013 -17640> 2013-04-03 10:37:24.624713 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427308) queue_proposal bl 654 bytes; ctx = 0x1b91f20 -17639> 2013-04-03 10:37:24.624721 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427308) propose_queued 427309 654 bytes -17638> 2013-04-03 10:37:24.627726 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427308 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4a00 -17637> 2013-04-03 10:37:24.669623 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113258 ==== paxos(accept lc 427308 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (342606253 0 0) 0x2ab4a00 con 0x208a840 -17636> 2013-04-03 10:37:24.672083 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427309 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4c80 -17635> 2013-04-03 10:37:24.672117 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427309 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29b4280 -17634> 2013-04-03 10:37:24.672183 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29012 update_from_paxos applying incremental 29013 -17633> 2013-04-03 10:37:24.672202 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 v29013: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2351B/s wr, 0op/s -17632> 2013-04-03 10:37:24.672221 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 send_pg_creates to 0 pgs -17631> 2013-04-03 10:37:24.672226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 update_logger -17630> 2013-04-03 10:37:24.672236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 create_pending v 29014 -17629> 2013-04-03 10:37:24.672245 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 _updated_stats for osd.1 10.252.24.25:6805/3572 -17628> 2013-04-03 10:37:24.672251 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 20) v1 tid 17498) v2 -- ?+0 0x20adb40 con 0x208a840 -17627> 2013-04-03 10:37:24.672262 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 check_osd_map already seen 602 >= 602 -17626> 2013-04-03 10:37:24.672265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 update_logger -17625> 2013-04-03 10:37:24.672275 7fcdffcdb700 0 log [INF] : pgmap v29013: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2351B/s wr, 0op/s -17624> 2013-04-03 10:37:24.672283 7fcdffcdb700 10 send_log to self -17623> 2013-04-03 10:37:24.672284 7fcdffcdb700 10 log_queue is 1 last_log 75 sent 74 num 1 unsent 1 sending 1 -17622> 2013-04-03 10:37:24.672285 7fcdffcdb700 10 will send 2013-04-03 10:37:24.672283 mon.0 10.252.24.25:6789/0 75 : [INF] pgmap v29013: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2351B/s wr, 0op/s -17621> 2013-04-03 10:37:24.672292 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287e400 -17620> 2013-04-03 10:37:24.672341 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos -17619> 2013-04-03 10:37:24.672362 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394016 -17618> 2013-04-03 10:37:24.672379 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17617> 2013-04-03 10:37:24.672401 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287e400 con 0x1bac160 -17616> 2013-04-03 10:37:24.672423 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos -17615> 2013-04-03 10:37:24.672439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394016 -17614> 2013-04-03 10:37:24.672447 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -17613> 2013-04-03 10:37:24.672457 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 preprocess_log log(1 entries) v1 from mon.0 -17612> 2013-04-03 10:37:24.672479 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -17611> 2013-04-03 10:37:24.672491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 prepare_log log(1 entries) v1 from mon.0 -17610> 2013-04-03 10:37:24.672502 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 logging 2013-04-03 10:37:24.672283 mon.0 10.252.24.25:6789/0 75 : [INF] pgmap v29013: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2351B/s wr, 0op/s -17609> 2013-04-03 10:37:24.683294 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113259 ==== paxos(lease_ack lc 427309 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (2220510838 0 0) 0x29b4280 con 0x208a840 -17608> 2013-04-03 10:37:24.683328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos -17607> 2013-04-03 10:37:24.683345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 update_from_paxos version 394016 summary v 394016 -17606> 2013-04-03 10:37:24.683360 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17605> 2013-04-03 10:37:24.927878 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113260 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 59 v187) v2) to leader v1 ==== 560+0+0 (3489783482 0 0) 0x2a83840 con 0x208a840 -17604> 2013-04-03 10:37:24.927900 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -17603> 2013-04-03 10:37:24.927911 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83580 from 10.252.24.88:6789/0 -17602> 2013-04-03 10:37:24.927962 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 59 v187) v2 from mds.0 10.252.24.25:6802/3144 -17601> 2013-04-03 10:37:24.927975 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 59 v187) v2 tid 17499) v2 -- ?+0 0x24f8900 con 0x208a840 -17600> 2013-04-03 10:37:25.381459 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 57 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21715 v187) v2 ==== 289+0+0 (980968234 0 0) 0x273b600 con 0x200d420 -17599> 2013-04-03 10:37:25.381530 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21715 v187) v2 from mds.? 10.252.24.88:6800/17213 -17598> 2013-04-03 10:37:25.381543 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21715 v187) v2 -- ?+0 0x2a83840 con 0x200d420 -17597> 2013-04-03 10:37:25.669814 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394016 encode_pending v394017 -17596> 2013-04-03 10:37:25.669842 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427309) queue_proposal bl 443 bytes; ctx = 0x1b91720 -17595> 2013-04-03 10:37:25.669853 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427309) propose_queued 427310 443 bytes -17594> 2013-04-03 10:37:25.670774 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427309 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4780 -17593> 2013-04-03 10:37:25.700035 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113261 ==== paxos(accept lc 427309 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (151724566 0 0) 0x2ab4780 con 0x208a840 -17592> 2013-04-03 10:37:25.704739 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427310 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4280 -17591> 2013-04-03 10:37:25.704773 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427310 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2ab4a00 -17590> 2013-04-03 10:37:25.704813 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos -17589> 2013-04-03 10:37:25.704838 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394016 -17588> 2013-04-03 10:37:25.704853 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos latest full 393798 -17587> 2013-04-03 10:37:25.704870 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos applying incremental log 394017 2013-04-03 10:37:24.672283 mon.0 10.252.24.25:6789/0 75 : [INF] pgmap v29013: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2351B/s wr, 0op/s -17586> 2013-04-03 10:37:25.704915 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 check_subs -17585> 2013-04-03 10:37:25.704947 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 create_pending v 394018 -17584> 2013-04-03 10:37:25.704970 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 _updated_log for mon.0 10.252.24.25:6789/0 -17583> 2013-04-03 10:37:25.704983 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 75) v1 -- ?+0 0x28a58c0 con 0x1bac160 -17582> 2013-04-03 10:37:25.705033 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos -17581> 2013-04-03 10:37:25.705050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 -17580> 2013-04-03 10:37:25.705068 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17579> 2013-04-03 10:37:25.705089 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 75) v1 ==== 0+0+0 (0 0 0) 0x28a58c0 con 0x1bac160 -17578> 2013-04-03 10:37:25.705097 7fcdffcdb700 10 handle_log_ack log(last 75) v1 -17577> 2013-04-03 10:37:25.705099 7fcdffcdb700 10 logged 2013-04-03 10:37:24.672283 mon.0 10.252.24.25:6789/0 75 : [INF] pgmap v29013: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2351B/s wr, 0op/s -17576> 2013-04-03 10:37:25.720730 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113262 ==== paxos(lease_ack lc 427310 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (670471769 0 0) 0x2ab4a00 con 0x208a840 -17575> 2013-04-03 10:37:25.720766 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos -17574> 2013-04-03 10:37:25.720783 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 -17573> 2013-04-03 10:37:25.720798 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17572> 2013-04-03 10:37:26.867732 7fcdfe8d5700 2 -- 10.252.24.25:6789/0 >> 10.252.24.30:6789/0 pipe(0x29b4500 sd=24 :6789 s=4 pgs=0 cs=0 l=0).fault 0: Success -17571> 2013-04-03 10:37:27.581946 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 58 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140345 v187) v2 ==== 289+0+0 (775900232 0 0) 0x2a512c0 con 0x2118420 -17570> 2013-04-03 10:37:27.582016 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140345 v187) v2 from mds.? 10.252.58.31:6800/10908 -17569> 2013-04-03 10:37:27.582031 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140345 v187) v2 -- ?+0 0x273b600 con 0x2118420 -17568> 2013-04-03 10:37:28.317270 7fcdfe8d5700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x29b4500 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept sd=24 10.252.24.30:45658/0 -17567> 2013-04-03 10:37:28.317771 7fcdfe8d5700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 ms_verify_authorizer 10.252.24.30:6789/0 mon protocol 2 -17566> 2013-04-03 10:37:28.317905 7fcdfe8d5700 0 -- 10.252.24.25:6789/0 >> 10.252.24.30:6789/0 pipe(0x29b4500 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state connecting -17565> 2013-04-03 10:37:28.335889 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 check_down_pgs -17564> 2013-04-03 10:37:28.335918 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 v29013: 240 pgs: 240 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 2351B/s wr, 0op/s -17563> 2013-04-03 10:37:28.335946 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -17562> 2013-04-03 10:37:28.335971 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 e602: 5 osds: 5 up, 5 in -17561> 2013-04-03 10:37:28.336008 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -17560> 2013-04-03 10:37:28.336039 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos -17559> 2013-04-03 10:37:28.336058 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 -17558> 2013-04-03 10:37:28.336068 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 log -17557> 2013-04-03 10:37:28.336085 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17556> 2013-04-03 10:37:28.336104 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -17555> 2013-04-03 10:37:28.339744 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113263 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (424405211 0 0) 0x2a51840 con 0x208a840 -17554> 2013-04-03 10:37:28.339760 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -17553> 2013-04-03 10:37:28.339767 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -17552> 2013-04-03 10:37:28.339772 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start sync session already in progress for mon.1 10.252.24.30:6789/0 -17551> 2013-04-03 10:37:28.339776 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start ignore stray message -17550> 2013-04-03 10:37:28.704836 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427310 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2ab4500 -17549> 2013-04-03 10:37:28.705517 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113264 ==== paxos(lease_ack lc 427310 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (1781675495 0 0) 0x2ab4500 con 0x208a840 -17548> 2013-04-03 10:37:28.705567 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos -17547> 2013-04-03 10:37:28.705585 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 -17546> 2013-04-03 10:37:28.705601 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17545> 2013-04-03 10:37:28.928006 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113265 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 60 v187) v2) to leader v1 ==== 560+0+0 (4009542835 0 0) 0x2a832c0 con 0x208a840 -17544> 2013-04-03 10:37:28.928027 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -17543> 2013-04-03 10:37:28.928040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83000 from 10.252.24.88:6789/0 -17542> 2013-04-03 10:37:28.928090 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 60 v187) v2 from mds.0 10.252.24.25:6802/3144 -17541> 2013-04-03 10:37:28.928102 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 60 v187) v2 tid 17500) v2 -- ?+0 0x20adb40 con 0x208a840 -17540> 2013-04-03 10:37:29.381580 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 58 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21716 v187) v2 ==== 289+0+0 (3530645337 0 0) 0x273b340 con 0x200d420 -17539> 2013-04-03 10:37:29.381653 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21716 v187) v2 from mds.? 10.252.24.88:6800/17213 -17538> 2013-04-03 10:37:29.381668 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21716 v187) v2 -- ?+0 0x2a832c0 con 0x200d420 -17537> 2013-04-03 10:37:29.962081 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113266 ==== forward(osd_boot(osd.2 booted 0 v602) v3) to leader v1 ==== 898+0+0 (312247691 0 0) 0x2a51840 con 0x208a840 -17536> 2013-04-03 10:37:29.962101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -17535> 2013-04-03 10:37:29.962111 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20c5800 from 10.252.24.88:6789/0 -17534> 2013-04-03 10:37:29.962161 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 preprocess_query osd_boot(osd.2 booted 0 v602) v3 from osd.2 10.252.24.30:6801/5656 -17533> 2013-04-03 10:37:29.962173 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 preprocess_boot from osd.2 10.252.24.30:6801/5656 -17532> 2013-04-03 10:37:29.962178 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 prepare_update osd_boot(osd.2 booted 0 v602) v3 from osd.2 10.252.24.30:6801/5656 -17531> 2013-04-03 10:37:29.962181 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 prepare_boot from osd.2 10.252.24.30:6801/5656 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.2 eb9ff30e-cc75-49d3-8ef9-2f478e173371 e602 [101,602] lci=[601,602]) cluster_addr 10.252.24.30:6802/5656 hb_addr 10.252.24.30:6803/5656 -17530> 2013-04-03 10:37:29.962191 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 prepare_boot was up, first marking down osd.2 10.252.24.30:6800/1508 -17529> 2013-04-03 10:37:29.962195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 should_propose -17528> 2013-04-03 10:37:30.012342 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -17527> 2013-04-03 10:37:30.012401 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 min_last_epoch_clean 593 -17526> 2013-04-03 10:37:30.012454 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 encode_pending e 603 -17525> 2013-04-03 10:37:30.012458 7fce004dc700 2 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 osd.2 DOWN -17524> 2013-04-03 10:37:30.012486 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427310) queue_proposal bl 290 bytes; ctx = 0x1b91f00 -17523> 2013-04-03 10:37:30.012498 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427310) propose_queued 427311 290 bytes -17522> 2013-04-03 10:37:30.014238 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427310 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4280 -17521> 2013-04-03 10:37:30.050966 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113267 ==== paxos(accept lc 427310 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1046173714 0 0) 0x2ab4280 con 0x208a840 -17520> 2013-04-03 10:37:30.062914 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427311 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4500 -17519> 2013-04-03 10:37:30.062948 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427311 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2ab4a00 -17518> 2013-04-03 10:37:30.063015 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e602 update_from_paxos applying incremental 603 -17517> 2013-04-03 10:37:30.063059 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 e603: 5 osds: 4 up, 5 in -17516> 2013-04-03 10:37:30.063975 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 adding osd.2 to down_pending_out map -17515> 2013-04-03 10:37:30.064031 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 check_osd_map applying osdmap e603 to pg_map -17514> 2013-04-03 10:37:30.064050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 register_new_pgs checking pg pools for osdmap epoch 603, last_pg_scan 341 -17513> 2013-04-03 10:37:30.064055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 -17512> 2013-04-03 10:37:30.064066 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -17511> 2013-04-03 10:37:30.064069 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -17510> 2013-04-03 10:37:30.064072 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 -17509> 2013-04-03 10:37:30.064075 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 -17508> 2013-04-03 10:37:30.064078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 -17507> 2013-04-03 10:37:30.064083 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 -17506> 2013-04-03 10:37:30.064086 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 -17505> 2013-04-03 10:37:30.064088 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 -17504> 2013-04-03 10:37:30.064099 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs -17503> 2013-04-03 10:37:30.064101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 check_down_pgs -17502> 2013-04-03 10:37:30.064103 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.d stale with acting [2,0] -17501> 2013-04-03 10:37:30.064108 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 8.3 stale with acting [2,3] -17500> 2013-04-03 10:37:30.064111 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.9 stale with acting [2,3] -17499> 2013-04-03 10:37:30.064114 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.a stale with acting [2,3] -17498> 2013-04-03 10:37:30.064116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.b stale with acting [2,3] -17497> 2013-04-03 10:37:30.064120 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 7.0 stale with acting [2,3] -17496> 2013-04-03 10:37:30.064122 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 6.1 stale with acting [2,3] -17495> 2013-04-03 10:37:30.064125 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 5.2 stale with acting [2,3] -17494> 2013-04-03 10:37:30.064128 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 4.3 stale with acting [2,3] -17493> 2013-04-03 10:37:30.064130 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 3.4 stale with acting [2,3] -17492> 2013-04-03 10:37:30.064133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.5 stale with acting [2,3] -17491> 2013-04-03 10:37:30.064136 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.6 stale with acting [2,3] -17490> 2013-04-03 10:37:30.064138 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.7 stale with acting [2,3] -17489> 2013-04-03 10:37:30.064141 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 5.0 stale with acting [2,0] -17488> 2013-04-03 10:37:30.064147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 4.1 stale with acting [2,0] -17487> 2013-04-03 10:37:30.064150 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.4 stale with acting [2,0] -17486> 2013-04-03 10:37:30.064152 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.5 stale with acting [2,0] -17485> 2013-04-03 10:37:30.064155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 7.4 stale with acting [2,3] -17484> 2013-04-03 10:37:30.064158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 6.5 stale with acting [2,3] -17483> 2013-04-03 10:37:30.064161 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 5.6 stale with acting [2,3] -17482> 2013-04-03 10:37:30.064166 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 4.7 stale with acting [2,3] -17481> 2013-04-03 10:37:30.064169 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.1 stale with acting [2,4] -17480> 2013-04-03 10:37:30.064172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.2 stale with acting [2,4] -17479> 2013-04-03 10:37:30.064174 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.3 stale with acting [2,4] -17478> 2013-04-03 10:37:30.064177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 3.0 stale with acting [2,4] -17477> 2013-04-03 10:37:30.064179 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 3.2 stale with acting [2,0] -17476> 2013-04-03 10:37:30.064182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.3 stale with acting [2,0] -17475> 2013-04-03 10:37:30.064185 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 7.7 stale with acting [2,0] -17474> 2013-04-03 10:37:30.064189 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.3c stale with acting [2,3] -17473> 2013-04-03 10:37:30.064191 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.3e stale with acting [2,3] -17472> 2013-04-03 10:37:30.064194 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.3d stale with acting [2,3] -17471> 2013-04-03 10:37:30.064197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.39 stale with acting [2,0] -17470> 2013-04-03 10:37:30.064200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.3a stale with acting [2,0] -17469> 2013-04-03 10:37:30.064205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.3b stale with acting [2,0] -17468> 2013-04-03 10:37:30.064208 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.38 stale with acting [2,3] -17467> 2013-04-03 10:37:30.064211 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.3a stale with acting [2,3] -17466> 2013-04-03 10:37:30.064213 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.39 stale with acting [2,3] -17465> 2013-04-03 10:37:30.064217 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.34 stale with acting [2,1] -17464> 2013-04-03 10:37:30.064220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.36 stale with acting [2,1] -17463> 2013-04-03 10:37:30.064222 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.35 stale with acting [2,1] -17462> 2013-04-03 10:37:30.064226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.29 stale with acting [2,0] -17461> 2013-04-03 10:37:30.064228 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.2a stale with acting [2,0] -17460> 2013-04-03 10:37:30.064231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.2b stale with acting [2,0] -17459> 2013-04-03 10:37:30.064234 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.28 stale with acting [2,0] -17458> 2013-04-03 10:37:30.064236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.2a stale with acting [2,0] -17457> 2013-04-03 10:37:30.064239 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.28 stale with acting [2,1] -17456> 2013-04-03 10:37:30.064241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.29 stale with acting [2,1] -17455> 2013-04-03 10:37:30.064244 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.29 stale with acting [2,0] -17454> 2013-04-03 10:37:30.064247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.27 stale with acting [2,1] -17453> 2013-04-03 10:37:30.064250 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.21 stale with acting [2,3] -17452> 2013-04-03 10:37:30.064252 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.22 stale with acting [2,3] -17451> 2013-04-03 10:37:30.064255 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.23 stale with acting [2,3] -17450> 2013-04-03 10:37:30.064261 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.20 stale with acting [2,1] -17449> 2013-04-03 10:37:30.064264 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.22 stale with acting [2,1] -17448> 2013-04-03 10:37:30.064267 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.21 stale with acting [2,1] -17447> 2013-04-03 10:37:30.064270 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.1c stale with acting [2,3] -17446> 2013-04-03 10:37:30.064275 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.1b stale with acting [2,3] -17445> 2013-04-03 10:37:30.064278 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.19 stale with acting [2,3] -17444> 2013-04-03 10:37:30.064281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.18 stale with acting [2,3] -17443> 2013-04-03 10:37:30.064284 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.18 stale with acting [2,3] -17442> 2013-04-03 10:37:30.064287 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.1a stale with acting [2,3] -17441> 2013-04-03 10:37:30.064290 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.17 stale with acting [2,3] -17440> 2013-04-03 10:37:30.064292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.17 stale with acting [2,3] -17439> 2013-04-03 10:37:30.064295 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.16 stale with acting [2,3] -17438> 2013-04-03 10:37:30.064299 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.f stale with acting [2,3] -17437> 2013-04-03 10:37:30.064301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 1.e stale with acting [2,3] -17436> 2013-04-03 10:37:30.064304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.d stale with acting [2,3] -17435> 2013-04-03 10:37:30.064307 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 8.7 stale with acting [2,3] -17434> 2013-04-03 10:37:30.064310 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 0.e stale with acting [2,0] -17433> 2013-04-03 10:37:30.064312 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 2.c stale with acting [2,0] -17432> 2013-04-03 10:37:30.064315 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 marking pg 8.6 stale with acting [2,0] -17431> 2013-04-03 10:37:30.064370 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 encode_pending v 29014 -17430> 2013-04-03 10:37:30.064565 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427311) queue_proposal bl 26698 bytes; ctx = 0x1b91e20 -17429> 2013-04-03 10:37:30.064572 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427311) propose_queued 427312 26698 bytes -17428> 2013-04-03 10:37:30.067424 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427311 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4780 -17427> 2013-04-03 10:37:30.067442 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 send_pg_creates to 0 pgs -17426> 2013-04-03 10:37:30.067447 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 send_to_waiting 603 -17425> 2013-04-03 10:37:30.067450 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 share_map_with_random_osd no up osd on our session map -17424> 2013-04-03 10:37:30.067452 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 update_logger -17423> 2013-04-03 10:37:30.067472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 create_pending e 604 -17422> 2013-04-03 10:37:30.067475 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 remove_redundant_pg_temp -17421> 2013-04-03 10:37:30.067477 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 remove_down_pg_temp -17420> 2013-04-03 10:37:30.067514 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_query osd_boot(osd.2 booted 0 v602) v3 from osd.2 10.252.24.30:6801/5656 -17419> 2013-04-03 10:37:30.067523 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_boot from osd.2 10.252.24.30:6801/5656 -17418> 2013-04-03 10:37:30.067526 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_update osd_boot(osd.2 booted 0 v602) v3 from osd.2 10.252.24.30:6801/5656 -17417> 2013-04-03 10:37:30.067530 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_boot from osd.2 10.252.24.30:6801/5656 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.2 eb9ff30e-cc75-49d3-8ef9-2f478e173371 e602 [101,602] lci=[601,602]) cluster_addr 10.252.24.30:6802/5656 hb_addr 10.252.24.30:6803/5656 -17416> 2013-04-03 10:37:30.067540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 setting osd.2 uuid to eb9ff30e-cc75-49d3-8ef9-2f478e173371 -17415> 2013-04-03 10:37:30.067544 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 old osd_info: up_from 601 up_thru 601 down_at 603 last_clean_interval [593,599) lost_at 95 -17414> 2013-04-03 10:37:30.067546 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_boot osd.2 last_clean_interval [593,599) -> [601-602) -17413> 2013-04-03 10:37:30.067549 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 not laggy, new xi down_stamp 2013-04-03 10:37:30.012458 laggy_probability 0 laggy_interval 0 -17412> 2013-04-03 10:37:30.067560 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 should_propose -17411> 2013-04-03 10:37:30.067572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 update_logger -17410> 2013-04-03 10:37:30.067579 7fcdffcdb700 0 log [INF] : osdmap e603: 5 osds: 4 up, 5 in -17409> 2013-04-03 10:37:30.067580 7fcdffcdb700 10 send_log to self -17408> 2013-04-03 10:37:30.067581 7fcdffcdb700 10 log_queue is 1 last_log 76 sent 75 num 1 unsent 1 sending 1 -17407> 2013-04-03 10:37:30.067582 7fcdffcdb700 10 will send 2013-04-03 10:37:30.067580 mon.0 10.252.24.25:6789/0 76 : [INF] osdmap e603: 5 osds: 4 up, 5 in -17406> 2013-04-03 10:37:30.067588 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2200600 -17405> 2013-04-03 10:37:30.067616 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2200600 con 0x1bac160 -17404> 2013-04-03 10:37:30.067641 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos -17403> 2013-04-03 10:37:30.067658 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 -17402> 2013-04-03 10:37:30.067668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -17401> 2013-04-03 10:37:30.067679 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 preprocess_log log(1 entries) v1 from mon.0 -17400> 2013-04-03 10:37:30.067702 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -17399> 2013-04-03 10:37:30.067715 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 prepare_log log(1 entries) v1 from mon.0 -17398> 2013-04-03 10:37:30.068262 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 logging 2013-04-03 10:37:30.067580 mon.0 10.252.24.25:6789/0 76 : [INF] osdmap e603: 5 osds: 4 up, 5 in -17397> 2013-04-03 10:37:30.072903 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113268 ==== paxos(lease_ack lc 427311 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (595605361 0 0) 0x2ab4780 con 0x208a840 -17396> 2013-04-03 10:37:30.125035 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113269 ==== paxos(accept lc 427311 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1609460397 0 0) 0x2ab4a00 con 0x208a840 -17395> 2013-04-03 10:37:30.126197 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427312 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4780 -17394> 2013-04-03 10:37:30.126226 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427312 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2ab4280 -17393> 2013-04-03 10:37:30.126323 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29013 update_from_paxos applying incremental 29014 -17392> 2013-04-03 10:37:30.126915 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 v29014: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 355B/s wr, 0op/s -17391> 2013-04-03 10:37:30.126956 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 send_pg_creates to 0 pgs -17390> 2013-04-03 10:37:30.126959 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 update_logger -17389> 2013-04-03 10:37:30.126981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 create_pending v 29015 -17388> 2013-04-03 10:37:30.126991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 check_osd_map already seen 603 >= 603 -17387> 2013-04-03 10:37:30.126993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 update_logger -17386> 2013-04-03 10:37:30.127005 7fcdffcdb700 0 log [INF] : pgmap v29014: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 355B/s wr, 0op/s -17385> 2013-04-03 10:37:30.127006 7fcdffcdb700 10 send_log to self -17384> 2013-04-03 10:37:30.127007 7fcdffcdb700 10 log_queue is 2 last_log 77 sent 76 num 2 unsent 1 sending 1 -17383> 2013-04-03 10:37:30.127009 7fcdffcdb700 10 will send 2013-04-03 10:37:30.067580 mon.0 10.252.24.25:6789/0 76 : [INF] osdmap e603: 5 osds: 4 up, 5 in -17382> 2013-04-03 10:37:30.127017 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287ca00 -17381> 2013-04-03 10:37:30.127063 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos -17380> 2013-04-03 10:37:30.127084 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 -17379> 2013-04-03 10:37:30.127102 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17378> 2013-04-03 10:37:30.127122 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287ca00 con 0x1bac160 -17377> 2013-04-03 10:37:30.127142 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos -17376> 2013-04-03 10:37:30.127155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 -17375> 2013-04-03 10:37:30.127163 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -17374> 2013-04-03 10:37:30.127177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 preprocess_log log(1 entries) v1 from mon.0 -17373> 2013-04-03 10:37:30.127199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 nothing new -17372> 2013-04-03 10:37:30.227408 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113270 ==== paxos(lease_ack lc 427312 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (2510609526 0 0) 0x2ab4280 con 0x208a840 -17371> 2013-04-03 10:37:30.227477 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos -17370> 2013-04-03 10:37:30.227497 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 update_from_paxos version 394017 summary v 394017 -17369> 2013-04-03 10:37:30.227514 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17368> 2013-04-03 10:37:30.227532 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113271 ==== forward(osd_alive(want up_thru 603 have 603) v1) to leader v1 ==== 294+0+0 (3427024081 0 0) 0x2995840 con 0x208a840 -17367> 2013-04-03 10:37:30.227540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 -17366> 2013-04-03 10:37:30.227549 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x28a5700 from 10.252.24.88:6789/0 -17365> 2013-04-03 10:37:30.227572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_query osd_alive(want up_thru 603 have 603) v1 from osd.4 10.252.24.32:6800/13473 -17364> 2013-04-03 10:37:30.227580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_alive want up_thru 603 from osd.4 10.252.24.32:6800/13473 -17363> 2013-04-03 10:37:30.227585 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_update osd_alive(want up_thru 603 have 603) v1 from osd.4 10.252.24.32:6800/13473 -17362> 2013-04-03 10:37:30.227589 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_alive want up_thru 603 have 603 from osd.4 10.252.24.32:6800/13473 -17361> 2013-04-03 10:37:30.227593 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 should_propose -17360> 2013-04-03 10:37:30.227804 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113272 ==== forward(osd_alive(want up_thru 603 have 603) v1) to leader v1 ==== 294+0+0 (934262243 0 0) 0x29952c0 con 0x208a840 -17359> 2013-04-03 10:37:30.227820 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -17358> 2013-04-03 10:37:30.227827 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x28a5540 from 10.252.24.88:6789/0 -17357> 2013-04-03 10:37:30.227851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_query osd_alive(want up_thru 603 have 603) v1 from osd.3 10.252.24.88:6801/17538 -17356> 2013-04-03 10:37:30.227858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_alive want up_thru 603 from osd.3 10.252.24.88:6801/17538 -17355> 2013-04-03 10:37:30.227864 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_update osd_alive(want up_thru 603 have 603) v1 from osd.3 10.252.24.88:6801/17538 -17354> 2013-04-03 10:37:30.227867 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_alive want up_thru 603 have 603 from osd.3 10.252.24.88:6801/17538 -17353> 2013-04-03 10:37:30.227870 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 should_propose -17352> 2013-04-03 10:37:30.227881 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113273 ==== forward(osd_alive(want up_thru 603 have 603) v1) to leader v1 ==== 294+0+0 (2973224046 0 0) 0x2995580 con 0x208a840 -17351> 2013-04-03 10:37:30.227888 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -17350> 2013-04-03 10:37:30.227893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x28a5380 from 10.252.24.88:6789/0 -17349> 2013-04-03 10:37:30.227914 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_query osd_alive(want up_thru 603 have 603) v1 from osd.0 10.252.58.31:6802/2682 -17348> 2013-04-03 10:37:30.227919 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 preprocess_alive want up_thru 603 from osd.0 10.252.58.31:6802/2682 -17347> 2013-04-03 10:37:30.227922 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_update osd_alive(want up_thru 603 have 603) v1 from osd.0 10.252.58.31:6802/2682 -17346> 2013-04-03 10:37:30.227926 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 prepare_alive want up_thru 603 have 603 from osd.0 10.252.58.31:6802/2682 -17345> 2013-04-03 10:37:30.227929 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 should_propose -17344> 2013-04-03 10:37:30.393174 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2acdc80 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33051/0 -17343> 2013-04-03 10:37:30.393194 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acdc80 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33051/0) -17342> 2013-04-03 10:37:30.393206 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acdc80 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -17341> 2013-04-03 10:37:30.393426 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acdc80 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -17340> 2013-04-03 10:37:31.051165 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394017 encode_pending v394018 -17339> 2013-04-03 10:37:31.051201 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427312) queue_proposal bl 364 bytes; ctx = 0x1b91f10 -17338> 2013-04-03 10:37:31.051209 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427312) propose_queued 427313 364 bytes -17337> 2013-04-03 10:37:31.052771 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427312 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4000 -17336> 2013-04-03 10:37:31.052850 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 min_last_epoch_clean 593 -17335> 2013-04-03 10:37:31.052907 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 min_last_epoch_clean 593 -17334> 2013-04-03 10:37:31.052957 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 encode_pending e 604 -17333> 2013-04-03 10:37:31.052962 7fce004dc700 2 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 osd.2 UP 10.252.24.30:6801/5656 -17332> 2013-04-03 10:37:31.052996 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427289..427312) queue_proposal bl 767 bytes; ctx = 0x1b91d70 -17331> 2013-04-03 10:37:31.053003 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427289..427312) propose_new_value not active; proposal queued -17330> 2013-04-03 10:37:31.088892 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113274 ==== paxos(accept lc 427312 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1009195272 0 0) 0x2ab4000 con 0x208a840 -17329> 2013-04-03 10:37:31.097660 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427313 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4280 -17328> 2013-04-03 10:37:31.097679 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427313 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2ab4a00 -17327> 2013-04-03 10:37:31.097718 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos -17326> 2013-04-03 10:37:31.097743 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos version 394018 summary v 394017 -17325> 2013-04-03 10:37:31.097758 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos latest full 393798 -17324> 2013-04-03 10:37:31.097775 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos applying incremental log 394018 2013-04-03 10:37:30.067580 mon.0 10.252.24.25:6789/0 76 : [INF] osdmap e603: 5 osds: 4 up, 5 in -17323> 2013-04-03 10:37:31.097820 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 check_subs -17322> 2013-04-03 10:37:31.097851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 create_pending v 394019 -17321> 2013-04-03 10:37:31.097876 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 _updated_log for mon.0 10.252.24.25:6789/0 -17320> 2013-04-03 10:37:31.097889 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 76) v1 -- ?+0 0x28a58c0 con 0x1bac160 -17319> 2013-04-03 10:37:31.097913 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427313) propose_queued 427314 767 bytes -17318> 2013-04-03 10:37:31.098130 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427313 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4000 -17317> 2013-04-03 10:37:31.098170 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 76) v1 ==== 0+0+0 (0 0 0) 0x28a58c0 con 0x1bac160 -17316> 2013-04-03 10:37:31.098178 7fcdffcdb700 10 handle_log_ack log(last 76) v1 -17315> 2013-04-03 10:37:31.098180 7fcdffcdb700 10 logged 2013-04-03 10:37:30.067580 mon.0 10.252.24.25:6789/0 76 : [INF] osdmap e603: 5 osds: 4 up, 5 in -17314> 2013-04-03 10:37:31.109567 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113275 ==== paxos(lease_ack lc 427313 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (1388500982 0 0) 0x29b4000 con 0x208a840 -17313> 2013-04-03 10:37:31.123187 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113276 ==== paxos(accept lc 427313 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2361542036 0 0) 0x2ab4a00 con 0x208a840 -17312> 2013-04-03 10:37:31.123438 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427314 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4000 -17311> 2013-04-03 10:37:31.123469 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427314 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2ab4000 -17310> 2013-04-03 10:37:31.123532 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e603 update_from_paxos applying incremental 604 -17309> 2013-04-03 10:37:31.123582 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 e604: 5 osds: 5 up, 5 in -17308> 2013-04-03 10:37:31.124833 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 check_osd_map applying osdmap e604 to pg_map -17307> 2013-04-03 10:37:31.124863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 register_new_pgs checking pg pools for osdmap epoch 604, last_pg_scan 341 -17306> 2013-04-03 10:37:31.124867 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 -17305> 2013-04-03 10:37:31.124872 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -17304> 2013-04-03 10:37:31.124875 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -17303> 2013-04-03 10:37:31.124878 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 -17302> 2013-04-03 10:37:31.124881 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 -17301> 2013-04-03 10:37:31.124883 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 -17300> 2013-04-03 10:37:31.124886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 -17299> 2013-04-03 10:37:31.124889 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 -17298> 2013-04-03 10:37:31.124892 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 -17297> 2013-04-03 10:37:31.124903 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs -17296> 2013-04-03 10:37:31.124905 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 check_down_pgs -17295> 2013-04-03 10:37:31.124961 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 encode_pending v 29015 -17294> 2013-04-03 10:37:31.124989 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427314) queue_proposal bl 215 bytes; ctx = 0x1b91bd0 -17293> 2013-04-03 10:37:31.124995 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427314) propose_queued 427315 215 bytes -17292> 2013-04-03 10:37:31.125193 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427314 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787280 -17291> 2013-04-03 10:37:31.125210 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 send_pg_creates to 0 pgs -17290> 2013-04-03 10:37:31.125215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_to_waiting 604 -17289> 2013-04-03 10:37:31.125218 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 share_map_with_random_osd no up osd on our session map -17288> 2013-04-03 10:37:31.125225 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 update_logger -17287> 2013-04-03 10:37:31.125248 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 create_pending e 605 -17286> 2013-04-03 10:37:31.125251 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 remove_redundant_pg_temp -17285> 2013-04-03 10:37:31.125253 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 remove_down_pg_temp -17284> 2013-04-03 10:37:31.125271 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 _booted osd.2 10.252.24.30:6801/5656 w 0 from 602 -17283> 2013-04-03 10:37:31.125288 7fcdffcdb700 0 log [INF] : osd.2 10.252.24.30:6801/5656 boot -17282> 2013-04-03 10:37:31.125289 7fcdffcdb700 10 send_log to self -17281> 2013-04-03 10:37:31.125290 7fcdffcdb700 10 log_queue is 2 last_log 78 sent 76 num 2 unsent 2 sending 2 -17280> 2013-04-03 10:37:31.125291 7fcdffcdb700 10 will send 2013-04-03 10:37:30.127006 mon.0 10.252.24.25:6789/0 77 : [INF] pgmap v29014: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 355B/s wr, 0op/s -17279> 2013-04-03 10:37:31.125296 7fcdffcdb700 10 will send 2013-04-03 10:37:31.125289 mon.0 10.252.24.25:6789/0 78 : [INF] osd.2 10.252.24.30:6801/5656 boot -17278> 2013-04-03 10:37:31.125300 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2200600 -17277> 2013-04-03 10:37:31.125319 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_latest to osd.2 10.252.24.30:6801/5656 start 603 -17276> 2013-04-03 10:37:31.125326 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_incremental [603..604] to osd.2 10.252.24.30:6801/5656 -17275> 2013-04-03 10:37:31.125338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 build_incremental [603..604] -17274> 2013-04-03 10:37:31.125371 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(603..604 src has 103..604) v3 tid 17501) v2 -- ?+0 0x24f8900 con 0x208a840 -17273> 2013-04-03 10:37:31.125384 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 _reply_map 603 from osd.4 10.252.24.32:6800/13473 -17272> 2013-04-03 10:37:31.125402 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_latest to osd.4 10.252.24.32:6800/13473 start 603 -17271> 2013-04-03 10:37:31.125408 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_incremental [603..604] to osd.4 10.252.24.32:6800/13473 -17270> 2013-04-03 10:37:31.125419 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 build_incremental [603..604] -17269> 2013-04-03 10:37:31.125451 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(603..604 src has 103..604) v3 tid 17502) v2 -- ?+0 0x20fc240 con 0x208a840 -17268> 2013-04-03 10:37:31.125461 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 _reply_map 603 from osd.3 10.252.24.88:6801/17538 -17267> 2013-04-03 10:37:31.125484 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_latest to osd.3 10.252.24.88:6801/17538 start 603 -17266> 2013-04-03 10:37:31.125491 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_incremental [603..604] to osd.3 10.252.24.88:6801/17538 -17265> 2013-04-03 10:37:31.125502 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 build_incremental [603..604] -17264> 2013-04-03 10:37:31.125533 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(603..604 src has 103..604) v3 tid 17503) v2 -- ?+0 0x20fc6c0 con 0x208a840 -17263> 2013-04-03 10:37:31.125542 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 _reply_map 603 from osd.0 10.252.58.31:6802/2682 -17262> 2013-04-03 10:37:31.125560 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_latest to osd.0 10.252.58.31:6802/2682 start 603 -17261> 2013-04-03 10:37:31.125564 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 send_incremental [603..604] to osd.0 10.252.58.31:6802/2682 -17260> 2013-04-03 10:37:31.125575 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 build_incremental [603..604] -17259> 2013-04-03 10:37:31.125605 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(603..604 src has 103..604) v3 tid 17504) v2 -- ?+0 0x20fcb40 con 0x208a840 -17258> 2013-04-03 10:37:31.125615 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 update_logger -17257> 2013-04-03 10:37:31.125621 7fcdffcdb700 0 log [INF] : osdmap e604: 5 osds: 5 up, 5 in -17256> 2013-04-03 10:37:31.125622 7fcdffcdb700 10 send_log to self -17255> 2013-04-03 10:37:31.125623 7fcdffcdb700 10 log_queue is 3 last_log 79 sent 78 num 3 unsent 1 sending 1 -17254> 2013-04-03 10:37:31.125624 7fcdffcdb700 10 will send 2013-04-03 10:37:31.125289 mon.0 10.252.24.25:6789/0 78 : [INF] osd.2 10.252.24.30:6801/5656 boot -17253> 2013-04-03 10:37:31.125628 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2874400 -17252> 2013-04-03 10:37:31.125652 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2200600 con 0x1bac160 -17251> 2013-04-03 10:37:31.125675 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos -17250> 2013-04-03 10:37:31.125693 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos version 394018 summary v 394018 -17249> 2013-04-03 10:37:31.125702 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -17248> 2013-04-03 10:37:31.125713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 preprocess_log log(2 entries) v1 from mon.0 -17247> 2013-04-03 10:37:31.125739 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -17246> 2013-04-03 10:37:31.125751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 prepare_log log(2 entries) v1 from mon.0 -17245> 2013-04-03 10:37:31.125762 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 logging 2013-04-03 10:37:30.127006 mon.0 10.252.24.25:6789/0 77 : [INF] pgmap v29014: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 355B/s wr, 0op/s -17244> 2013-04-03 10:37:31.125781 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 logging 2013-04-03 10:37:31.125289 mon.0 10.252.24.25:6789/0 78 : [INF] osd.2 10.252.24.30:6801/5656 boot -17243> 2013-04-03 10:37:31.125820 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2874400 con 0x1bac160 -17242> 2013-04-03 10:37:31.125843 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos -17241> 2013-04-03 10:37:31.125858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos version 394018 summary v 394018 -17240> 2013-04-03 10:37:31.125866 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -17239> 2013-04-03 10:37:31.125876 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 preprocess_log log(1 entries) v1 from mon.0 -17238> 2013-04-03 10:37:31.125892 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 nothing new -17237> 2013-04-03 10:37:31.149461 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113277 ==== paxos(lease_ack lc 427314 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (1638691007 0 0) 0x29fdc80 con 0x208a840 -17236> 2013-04-03 10:37:31.187675 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113278 ==== paxos(accept lc 427314 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2737474002 0 0) 0x29e0000 con 0x208a840 -17235> 2013-04-03 10:37:31.188034 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427315 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fdc80 -17234> 2013-04-03 10:37:31.188061 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427315 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2ab4a00 -17233> 2013-04-03 10:37:31.188143 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29014 update_from_paxos applying incremental 29015 -17232> 2013-04-03 10:37:31.188155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 v29015: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -17231> 2013-04-03 10:37:31.188175 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 send_pg_creates to 0 pgs -17230> 2013-04-03 10:37:31.188178 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 update_logger -17229> 2013-04-03 10:37:31.188188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 create_pending v 29016 -17228> 2013-04-03 10:37:31.188197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 check_osd_map already seen 604 >= 604 -17227> 2013-04-03 10:37:31.188200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 update_logger -17226> 2013-04-03 10:37:31.188210 7fcdffcdb700 0 log [INF] : pgmap v29015: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -17225> 2013-04-03 10:37:31.188212 7fcdffcdb700 10 send_log to self -17224> 2013-04-03 10:37:31.188212 7fcdffcdb700 10 log_queue is 4 last_log 80 sent 78 num 4 unsent 2 sending 2 -17223> 2013-04-03 10:37:31.188214 7fcdffcdb700 10 will send 2013-04-03 10:37:31.125289 mon.0 10.252.24.25:6789/0 78 : [INF] osd.2 10.252.24.30:6801/5656 boot -17222> 2013-04-03 10:37:31.188221 7fcdffcdb700 10 will send 2013-04-03 10:37:31.125622 mon.0 10.252.24.25:6789/0 79 : [INF] osdmap e604: 5 osds: 5 up, 5 in -17221> 2013-04-03 10:37:31.188225 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2874400 -17220> 2013-04-03 10:37:31.188272 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos -17219> 2013-04-03 10:37:31.188292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos version 394018 summary v 394018 -17218> 2013-04-03 10:37:31.188311 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17217> 2013-04-03 10:37:31.188332 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2874400 con 0x1bac160 -17216> 2013-04-03 10:37:31.188353 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos -17215> 2013-04-03 10:37:31.188368 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos version 394018 summary v 394018 -17214> 2013-04-03 10:37:31.188378 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -17213> 2013-04-03 10:37:31.188389 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 preprocess_log log(2 entries) v1 from mon.0 -17212> 2013-04-03 10:37:31.188420 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -17211> 2013-04-03 10:37:31.188432 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 prepare_log log(2 entries) v1 from mon.0 -17210> 2013-04-03 10:37:31.188443 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 logging 2013-04-03 10:37:31.125289 mon.0 10.252.24.25:6789/0 78 : [INF] osd.2 10.252.24.30:6801/5656 boot -17209> 2013-04-03 10:37:31.188462 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 logging 2013-04-03 10:37:31.125622 mon.0 10.252.24.25:6789/0 79 : [INF] osdmap e604: 5 osds: 5 up, 5 in -17208> 2013-04-03 10:37:31.241171 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113279 ==== paxos(lease_ack lc 427315 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (3446551936 0 0) 0x2ab4a00 con 0x208a840 -17207> 2013-04-03 10:37:31.241211 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos -17206> 2013-04-03 10:37:31.241228 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 update_from_paxos version 394018 summary v 394018 -17205> 2013-04-03 10:37:31.241243 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17204> 2013-04-03 10:37:31.241478 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113280 ==== forward(osd_alive(want up_thru 604 have 604) v1) to leader v1 ==== 294+0+0 (1049711959 0 0) 0x2995dc0 con 0x208a840 -17203> 2013-04-03 10:37:31.241488 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -17202> 2013-04-03 10:37:31.241497 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x28a51c0 from 10.252.24.88:6789/0 -17201> 2013-04-03 10:37:31.241523 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_query osd_alive(want up_thru 604 have 604) v1 from osd.2 10.252.24.30:6801/5656 -17200> 2013-04-03 10:37:31.241530 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_alive want up_thru 604 from osd.2 10.252.24.30:6801/5656 -17199> 2013-04-03 10:37:31.241534 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_update osd_alive(want up_thru 604 have 604) v1 from osd.2 10.252.24.30:6801/5656 -17198> 2013-04-03 10:37:31.241538 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_alive want up_thru 604 have 604 from osd.2 10.252.24.30:6801/5656 -17197> 2013-04-03 10:37:31.241542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 should_propose -17196> 2013-04-03 10:37:31.241553 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113281 ==== forward(osd_alive(want up_thru 604 have 604) v1) to leader v1 ==== 294+0+0 (255790128 0 0) 0x273a580 con 0x208a840 -17195> 2013-04-03 10:37:31.241560 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -17194> 2013-04-03 10:37:31.241565 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2294e00 from 10.252.24.88:6789/0 -17193> 2013-04-03 10:37:31.241587 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_query osd_alive(want up_thru 604 have 604) v1 from osd.3 10.252.24.88:6801/17538 -17192> 2013-04-03 10:37:31.241592 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_alive want up_thru 604 from osd.3 10.252.24.88:6801/17538 -17191> 2013-04-03 10:37:31.241596 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_update osd_alive(want up_thru 604 have 604) v1 from osd.3 10.252.24.88:6801/17538 -17190> 2013-04-03 10:37:31.241599 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_alive want up_thru 604 have 604 from osd.3 10.252.24.88:6801/17538 -17189> 2013-04-03 10:37:31.241602 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 should_propose -17188> 2013-04-03 10:37:31.244227 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113282 ==== forward(osd_alive(want up_thru 604 have 604) v1) to leader v1 ==== 294+0+0 (4079896163 0 0) 0x273a2c0 con 0x208a840 -17187> 2013-04-03 10:37:31.244240 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -17186> 2013-04-03 10:37:31.244248 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2294540 from 10.252.24.88:6789/0 -17185> 2013-04-03 10:37:31.244279 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_query osd_alive(want up_thru 604 have 604) v1 from osd.1 10.252.24.25:6805/3572 -17184> 2013-04-03 10:37:31.244287 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_alive want up_thru 604 from osd.1 10.252.24.25:6805/3572 -17183> 2013-04-03 10:37:31.244291 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_update osd_alive(want up_thru 604 have 604) v1 from osd.1 10.252.24.25:6805/3572 -17182> 2013-04-03 10:37:31.244294 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_alive want up_thru 604 have 604 from osd.1 10.252.24.25:6805/3572 -17181> 2013-04-03 10:37:31.244298 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 should_propose -17180> 2013-04-03 10:37:31.249669 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113283 ==== forward(osd_alive(want up_thru 604 have 604) v1) to leader v1 ==== 294+0+0 (1805034545 0 0) 0x273a000 con 0x208a840 -17179> 2013-04-03 10:37:31.249686 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -17178> 2013-04-03 10:37:31.249694 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2294a80 from 10.252.24.88:6789/0 -17177> 2013-04-03 10:37:31.249726 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_query osd_alive(want up_thru 604 have 604) v1 from osd.0 10.252.58.31:6802/2682 -17176> 2013-04-03 10:37:31.249733 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 preprocess_alive want up_thru 604 from osd.0 10.252.58.31:6802/2682 -17175> 2013-04-03 10:37:31.249736 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_update osd_alive(want up_thru 604 have 604) v1 from osd.0 10.252.58.31:6802/2682 -17174> 2013-04-03 10:37:31.249741 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 prepare_alive want up_thru 604 have 604 from osd.0 10.252.58.31:6802/2682 -17173> 2013-04-03 10:37:31.249744 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 should_propose -17172> 2013-04-03 10:37:31.582093 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 59 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140346 v187) v2 ==== 289+0+0 (1267502543 0 0) 0x2a51000 con 0x2118420 -17171> 2013-04-03 10:37:31.582148 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140346 v187) v2 from mds.? 10.252.58.31:6800/10908 -17170> 2013-04-03 10:37:31.582161 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140346 v187) v2 -- ?+0 0x273a000 con 0x2118420 -17169> 2013-04-03 10:37:32.123385 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394018 encode_pending v394019 -17168> 2013-04-03 10:37:32.123416 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427315) queue_proposal bl 875 bytes; ctx = 0x1b91db0 -17167> 2013-04-03 10:37:32.123423 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427315) propose_queued 427316 875 bytes -17166> 2013-04-03 10:37:32.124835 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427315 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787280 -17165> 2013-04-03 10:37:32.171900 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113284 ==== paxos(accept lc 427315 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (194718114 0 0) 0x2787280 con 0x208a840 -17164> 2013-04-03 10:37:32.172794 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427316 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4a00 -17163> 2013-04-03 10:37:32.172825 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427316 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29e0000 -17162> 2013-04-03 10:37:32.172866 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos -17161> 2013-04-03 10:37:32.172902 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos version 394019 summary v 394018 -17160> 2013-04-03 10:37:32.172917 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos latest full 393798 -17159> 2013-04-03 10:37:32.172936 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos applying incremental log 394019 2013-04-03 10:37:30.127006 mon.0 10.252.24.25:6789/0 77 : [INF] pgmap v29014: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 355B/s wr, 0op/s -17158> 2013-04-03 10:37:32.172965 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos applying incremental log 394019 2013-04-03 10:37:31.125289 mon.0 10.252.24.25:6789/0 78 : [INF] osd.2 10.252.24.30:6801/5656 boot -17157> 2013-04-03 10:37:32.172985 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos applying incremental log 394019 2013-04-03 10:37:31.125622 mon.0 10.252.24.25:6789/0 79 : [INF] osdmap e604: 5 osds: 5 up, 5 in -17156> 2013-04-03 10:37:32.173021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 check_subs -17155> 2013-04-03 10:37:32.173055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 create_pending v 394020 -17154> 2013-04-03 10:37:32.173078 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 _updated_log for mon.0 10.252.24.25:6789/0 -17153> 2013-04-03 10:37:32.173091 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 78) v1 -- ?+0 0x28a5380 con 0x1bac160 -17152> 2013-04-03 10:37:32.173099 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 _updated_log for mon.0 10.252.24.25:6789/0 -17151> 2013-04-03 10:37:32.173114 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 79) v1 -- ?+0 0x28a5540 con 0x1bac160 -17150> 2013-04-03 10:37:32.173163 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos -17149> 2013-04-03 10:37:32.173179 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos version 394019 summary v 394019 -17148> 2013-04-03 10:37:32.173197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17147> 2013-04-03 10:37:32.173218 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 78) v1 ==== 0+0+0 (0 0 0) 0x28a5380 con 0x1bac160 -17146> 2013-04-03 10:37:32.173226 7fcdffcdb700 10 handle_log_ack log(last 78) v1 -17145> 2013-04-03 10:37:32.173229 7fcdffcdb700 10 logged 2013-04-03 10:37:30.127006 mon.0 10.252.24.25:6789/0 77 : [INF] pgmap v29014: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 355B/s wr, 0op/s -17144> 2013-04-03 10:37:32.173235 7fcdffcdb700 10 logged 2013-04-03 10:37:31.125289 mon.0 10.252.24.25:6789/0 78 : [INF] osd.2 10.252.24.30:6801/5656 boot -17143> 2013-04-03 10:37:32.173241 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 79) v1 ==== 0+0+0 (0 0 0) 0x28a5540 con 0x1bac160 -17142> 2013-04-03 10:37:32.173246 7fcdffcdb700 10 handle_log_ack log(last 79) v1 -17141> 2013-04-03 10:37:32.173247 7fcdffcdb700 10 logged 2013-04-03 10:37:31.125622 mon.0 10.252.24.25:6789/0 79 : [INF] osdmap e604: 5 osds: 5 up, 5 in -17140> 2013-04-03 10:37:32.187831 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 min_last_epoch_clean 593 -17139> 2013-04-03 10:37:32.187892 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 min_last_epoch_clean 593 -17138> 2013-04-03 10:37:32.188585 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 encode_pending e 605 -17137> 2013-04-03 10:37:32.188622 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427316) queue_proposal bl 317 bytes; ctx = 0x1b91ef0 -17136> 2013-04-03 10:37:32.188631 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427316) propose_queued 427317 317 bytes -17135> 2013-04-03 10:37:32.197028 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427316 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787a00 -17134> 2013-04-03 10:37:32.197912 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113285 ==== paxos(lease_ack lc 427316 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (460636104 0 0) 0x2787a00 con 0x208a840 -17133> 2013-04-03 10:37:32.211559 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113286 ==== paxos(accept lc 427316 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2046367002 0 0) 0x29e0000 con 0x208a840 -17132> 2013-04-03 10:37:32.211837 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427317 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787a00 -17131> 2013-04-03 10:37:32.211868 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427317 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2787280 -17130> 2013-04-03 10:37:32.211935 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e604 update_from_paxos applying incremental 605 -17129> 2013-04-03 10:37:32.211981 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -17128> 2013-04-03 10:37:32.213128 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 check_osd_map applying osdmap e605 to pg_map -17127> 2013-04-03 10:37:32.213151 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 register_new_pgs checking pg pools for osdmap epoch 605, last_pg_scan 341 -17126> 2013-04-03 10:37:32.213158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 -17125> 2013-04-03 10:37:32.213163 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -17124> 2013-04-03 10:37:32.213165 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -17123> 2013-04-03 10:37:32.213168 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 -17122> 2013-04-03 10:37:32.213171 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 -17121> 2013-04-03 10:37:32.213174 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 -17120> 2013-04-03 10:37:32.213177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 -17119> 2013-04-03 10:37:32.213180 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 -17118> 2013-04-03 10:37:32.213182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 -17117> 2013-04-03 10:37:32.213192 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs -17116> 2013-04-03 10:37:32.213195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 check_down_pgs -17115> 2013-04-03 10:37:32.213256 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 encode_pending v 29016 -17114> 2013-04-03 10:37:32.213282 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427317) queue_proposal bl 215 bytes; ctx = 0x1b91b90 -17113> 2013-04-03 10:37:32.213289 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427317) propose_queued 427318 215 bytes -17112> 2013-04-03 10:37:32.213503 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427317 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2a2b780 -17111> 2013-04-03 10:37:32.213535 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 send_pg_creates to 0 pgs -17110> 2013-04-03 10:37:32.213545 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_to_waiting 605 -17109> 2013-04-03 10:37:32.213549 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 share_map_with_random_osd no up osd on our session map -17108> 2013-04-03 10:37:32.213551 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 update_logger -17107> 2013-04-03 10:37:32.213576 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 create_pending e 606 -17106> 2013-04-03 10:37:32.213579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 remove_redundant_pg_temp -17105> 2013-04-03 10:37:32.213581 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 remove_down_pg_temp -17104> 2013-04-03 10:37:32.213600 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 _reply_map 604 from osd.2 10.252.24.30:6801/5656 -17103> 2013-04-03 10:37:32.213618 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_latest to osd.2 10.252.24.30:6801/5656 start 604 -17102> 2013-04-03 10:37:32.213622 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_incremental [604..605] to osd.2 10.252.24.30:6801/5656 -17101> 2013-04-03 10:37:32.213633 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 build_incremental [604..605] -17100> 2013-04-03 10:37:32.213665 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(604..605 src has 104..605) v3 tid 17505) v2 -- ?+0 0x20fc000 con 0x208a840 -17099> 2013-04-03 10:37:32.213677 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 _reply_map 604 from osd.3 10.252.24.88:6801/17538 -17098> 2013-04-03 10:37:32.213696 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_latest to osd.3 10.252.24.88:6801/17538 start 604 -17097> 2013-04-03 10:37:32.213700 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_incremental [604..605] to osd.3 10.252.24.88:6801/17538 -17096> 2013-04-03 10:37:32.213710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 build_incremental [604..605] -17095> 2013-04-03 10:37:32.213741 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(604..605 src has 104..605) v3 tid 17506) v2 -- ?+0 0x24f86c0 con 0x208a840 -17094> 2013-04-03 10:37:32.213750 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 _reply_map 604 from osd.1 10.252.24.25:6805/3572 -17093> 2013-04-03 10:37:32.213766 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_latest to osd.1 10.252.24.25:6805/3572 start 604 -17092> 2013-04-03 10:37:32.213770 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_incremental [604..605] to osd.1 10.252.24.25:6805/3572 -17091> 2013-04-03 10:37:32.213780 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 build_incremental [604..605] -17090> 2013-04-03 10:37:32.213821 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(604..605 src has 104..605) v3 tid 17507) v2 -- ?+0 0x24f8900 con 0x208a840 -17089> 2013-04-03 10:37:32.213831 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 _reply_map 604 from osd.0 10.252.58.31:6802/2682 -17088> 2013-04-03 10:37:32.213859 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_latest to osd.0 10.252.58.31:6802/2682 start 604 -17087> 2013-04-03 10:37:32.213864 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 send_incremental [604..605] to osd.0 10.252.58.31:6802/2682 -17086> 2013-04-03 10:37:32.213874 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 build_incremental [604..605] -17085> 2013-04-03 10:37:32.213909 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(604..605 src has 104..605) v3 tid 17508) v2 -- ?+0 0x24f8d80 con 0x208a840 -17084> 2013-04-03 10:37:32.213918 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 update_logger -17083> 2013-04-03 10:37:32.213926 7fcdffcdb700 0 log [INF] : osdmap e605: 5 osds: 5 up, 5 in -17082> 2013-04-03 10:37:32.213927 7fcdffcdb700 10 send_log to self -17081> 2013-04-03 10:37:32.213928 7fcdffcdb700 10 log_queue is 2 last_log 81 sent 79 num 2 unsent 2 sending 2 -17080> 2013-04-03 10:37:32.213930 7fcdffcdb700 10 will send 2013-04-03 10:37:31.188212 mon.0 10.252.24.25:6789/0 80 : [INF] pgmap v29015: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -17079> 2013-04-03 10:37:32.213935 7fcdffcdb700 10 will send 2013-04-03 10:37:32.213927 mon.0 10.252.24.25:6789/0 81 : [INF] osdmap e605: 5 osds: 5 up, 5 in -17078> 2013-04-03 10:37:32.213938 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x21fe800 -17077> 2013-04-03 10:37:32.213967 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x21fe800 con 0x1bac160 -17076> 2013-04-03 10:37:32.213988 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos -17075> 2013-04-03 10:37:32.214002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos version 394019 summary v 394019 -17074> 2013-04-03 10:37:32.214010 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -17073> 2013-04-03 10:37:32.214020 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 preprocess_log log(2 entries) v1 from mon.0 -17072> 2013-04-03 10:37:32.214047 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -17071> 2013-04-03 10:37:32.214059 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 prepare_log log(2 entries) v1 from mon.0 -17070> 2013-04-03 10:37:32.214070 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 logging 2013-04-03 10:37:31.188212 mon.0 10.252.24.25:6789/0 80 : [INF] pgmap v29015: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -17069> 2013-04-03 10:37:32.214091 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 logging 2013-04-03 10:37:32.213927 mon.0 10.252.24.25:6789/0 81 : [INF] osdmap e605: 5 osds: 5 up, 5 in -17068> 2013-04-03 10:37:32.223920 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113287 ==== paxos(lease_ack lc 427317 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (1103008858 0 0) 0x2acdc80 con 0x208a840 -17067> 2013-04-03 10:37:32.263702 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113288 ==== paxos(accept lc 427317 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (260903928 0 0) 0x2a2b000 con 0x208a840 -17066> 2013-04-03 10:37:32.274535 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427318 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2acdc80 -17065> 2013-04-03 10:37:32.274554 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427318 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29e0000 -17064> 2013-04-03 10:37:32.274625 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29015 update_from_paxos applying incremental 29016 -17063> 2013-04-03 10:37:32.274634 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 v29016: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -17062> 2013-04-03 10:37:32.274652 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 send_pg_creates to 0 pgs -17061> 2013-04-03 10:37:32.274655 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 update_logger -17060> 2013-04-03 10:37:32.274665 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 create_pending v 29017 -17059> 2013-04-03 10:37:32.274674 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 check_osd_map already seen 605 >= 605 -17058> 2013-04-03 10:37:32.274676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 update_logger -17057> 2013-04-03 10:37:32.274686 7fcdffcdb700 0 log [INF] : pgmap v29016: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -17056> 2013-04-03 10:37:32.274687 7fcdffcdb700 10 send_log to self -17055> 2013-04-03 10:37:32.274688 7fcdffcdb700 10 log_queue is 3 last_log 82 sent 81 num 3 unsent 1 sending 1 -17054> 2013-04-03 10:37:32.274690 7fcdffcdb700 10 will send 2013-04-03 10:37:32.213927 mon.0 10.252.24.25:6789/0 81 : [INF] osdmap e605: 5 osds: 5 up, 5 in -17053> 2013-04-03 10:37:32.274696 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2200600 -17052> 2013-04-03 10:37:32.274740 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos -17051> 2013-04-03 10:37:32.274759 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos version 394019 summary v 394019 -17050> 2013-04-03 10:37:32.274779 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17049> 2013-04-03 10:37:32.274800 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2200600 con 0x1bac160 -17048> 2013-04-03 10:37:32.274822 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos -17047> 2013-04-03 10:37:32.274837 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos version 394019 summary v 394019 -17046> 2013-04-03 10:37:32.274847 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -17045> 2013-04-03 10:37:32.274858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 preprocess_log log(1 entries) v1 from mon.0 -17044> 2013-04-03 10:37:32.274879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 nothing new -17043> 2013-04-03 10:37:32.301443 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113289 ==== paxos(lease_ack lc 427318 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (3199051983 0 0) 0x29e0000 con 0x208a840 -17042> 2013-04-03 10:37:32.301487 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos -17041> 2013-04-03 10:37:32.301504 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 update_from_paxos version 394019 summary v 394019 -17040> 2013-04-03 10:37:32.301519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17039> 2013-04-03 10:37:32.928106 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113290 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 61 v187) v2) to leader v1 ==== 560+0+0 (4064022815 0 0) 0x273b600 con 0x208a840 -17038> 2013-04-03 10:37:32.928127 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -17037> 2013-04-03 10:37:32.928138 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20e4340 from 10.252.24.88:6789/0 -17036> 2013-04-03 10:37:32.928182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 61 v187) v2 from mds.0 10.252.24.25:6802/3144 -17035> 2013-04-03 10:37:32.928195 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 61 v187) v2 tid 17509) v2 -- ?+0 0x2b26d80 con 0x208a840 -17034> 2013-04-03 10:37:33.211759 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394019 encode_pending v394020 -17033> 2013-04-03 10:37:33.211789 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427318) queue_proposal bl 651 bytes; ctx = 0x1b91bf0 -17032> 2013-04-03 10:37:33.211797 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427318) propose_queued 427319 651 bytes -17031> 2013-04-03 10:37:33.213297 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427318 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787c80 -17030> 2013-04-03 10:37:33.236912 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113291 ==== paxos(accept lc 427318 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (979320882 0 0) 0x2787c80 con 0x208a840 -17029> 2013-04-03 10:37:33.263814 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427319 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0000 -17028> 2013-04-03 10:37:33.263845 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427319 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2a2b000 -17027> 2013-04-03 10:37:33.263888 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos -17026> 2013-04-03 10:37:33.263926 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos version 394020 summary v 394019 -17025> 2013-04-03 10:37:33.263940 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos latest full 393798 -17024> 2013-04-03 10:37:33.263957 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos applying incremental log 394020 2013-04-03 10:37:31.188212 mon.0 10.252.24.25:6789/0 80 : [INF] pgmap v29015: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -17023> 2013-04-03 10:37:33.263986 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos applying incremental log 394020 2013-04-03 10:37:32.213927 mon.0 10.252.24.25:6789/0 81 : [INF] osdmap e605: 5 osds: 5 up, 5 in -17022> 2013-04-03 10:37:33.264023 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 check_subs -17021> 2013-04-03 10:37:33.264055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 create_pending v 394021 -17020> 2013-04-03 10:37:33.264077 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 _updated_log for mon.0 10.252.24.25:6789/0 -17019> 2013-04-03 10:37:33.264091 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 81) v1 -- ?+0 0x2294a80 con 0x1bac160 -17018> 2013-04-03 10:37:33.264142 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos -17017> 2013-04-03 10:37:33.264158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos version 394020 summary v 394020 -17016> 2013-04-03 10:37:33.264461 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17015> 2013-04-03 10:37:33.264488 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 81) v1 ==== 0+0+0 (0 0 0) 0x2294a80 con 0x1bac160 -17014> 2013-04-03 10:37:33.264496 7fcdffcdb700 10 handle_log_ack log(last 81) v1 -17013> 2013-04-03 10:37:33.264499 7fcdffcdb700 10 logged 2013-04-03 10:37:31.188212 mon.0 10.252.24.25:6789/0 80 : [INF] pgmap v29015: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -17012> 2013-04-03 10:37:33.264505 7fcdffcdb700 10 logged 2013-04-03 10:37:32.213927 mon.0 10.252.24.25:6789/0 81 : [INF] osdmap e605: 5 osds: 5 up, 5 in -17011> 2013-04-03 10:37:33.311018 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113292 ==== paxos(lease_ack lc 427319 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (442537853 0 0) 0x2a2b000 con 0x208a840 -17010> 2013-04-03 10:37:33.311063 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos -17009> 2013-04-03 10:37:33.311081 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos version 394020 summary v 394020 -17008> 2013-04-03 10:37:33.311098 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -17007> 2013-04-03 10:37:33.313827 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113293 ==== forward(pg_stats(71 pgs tid 1 v 605) v1) to leader v1 ==== 26859+0+0 (1983336556 0 0) 0x2a51000 con 0x208a840 -17006> 2013-04-03 10:37:33.313840 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -17005> 2013-04-03 10:37:33.313849 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26d80 from 10.252.24.88:6789/0 -17004> 2013-04-03 10:37:33.313880 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 preprocess_query pg_stats(71 pgs tid 1 v 605) v1 from osd.2 10.252.24.30:6801/5656 -17003> 2013-04-03 10:37:33.313888 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 prepare_update pg_stats(71 pgs tid 1 v 605) v1 from osd.2 10.252.24.30:6801/5656 -17002> 2013-04-03 10:37:33.313893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 prepare_pg_stats pg_stats(71 pgs tid 1 v 605) v1 from osd.2 -17001> 2013-04-03 10:37:33.313899 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,3]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -17000> 2013-04-03 10:37:33.336219 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 check_down_pgs -16999> 2013-04-03 10:37:33.336255 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 v29016: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -16998> 2013-04-03 10:37:33.336284 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -16997> 2013-04-03 10:37:33.336306 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -16996> 2013-04-03 10:37:33.336344 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -16995> 2013-04-03 10:37:33.336374 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos -16994> 2013-04-03 10:37:33.336392 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos version 394020 summary v 394020 -16993> 2013-04-03 10:37:33.336400 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 log -16992> 2013-04-03 10:37:33.336415 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -16991> 2013-04-03 10:37:33.336432 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -16990> 2013-04-03 10:37:33.381633 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 59 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21717 v187) v2 ==== 289+0+0 (4045398052 0 0) 0x2a84600 con 0x200d420 -16989> 2013-04-03 10:37:33.381680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21717 v187) v2 from mds.? 10.252.24.88:6800/17213 -16988> 2013-04-03 10:37:33.381691 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21717 v187) v2 -- ?+0 0x2a51000 con 0x200d420 -16987> 2013-04-03 10:37:34.237113 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 encode_pending v 29017 -16986> 2013-04-03 10:37:34.237555 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427319) queue_proposal bl 26756 bytes; ctx = 0x1b91e10 -16985> 2013-04-03 10:37:34.237570 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427319) propose_queued 427320 26756 bytes -16984> 2013-04-03 10:37:34.239151 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427319 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787500 -16983> 2013-04-03 10:37:34.580757 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113294 ==== paxos(accept lc 427319 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1532187913 0 0) 0x2787500 con 0x208a840 -16982> 2013-04-03 10:37:34.582039 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427320 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2a2b000 -16981> 2013-04-03 10:37:34.582063 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427320 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2787c80 -16980> 2013-04-03 10:37:34.582152 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29016 update_from_paxos applying incremental 29017 -16979> 2013-04-03 10:37:34.582336 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 v29017: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 23690B/s wr, 0op/s -16978> 2013-04-03 10:37:34.582367 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 send_pg_creates to 0 pgs -16977> 2013-04-03 10:37:34.582371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 update_logger -16976> 2013-04-03 10:37:34.582388 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 create_pending v 29018 -16975> 2013-04-03 10:37:34.582402 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 _updated_stats for osd.2 10.252.24.30:6801/5656 -16974> 2013-04-03 10:37:34.582409 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(71 pgs tid 1) v1 tid 17510) v2 -- ?+0 0x24f8d80 con 0x208a840 -16973> 2013-04-03 10:37:34.582432 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 check_osd_map already seen 605 >= 605 -16972> 2013-04-03 10:37:34.582436 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 update_logger -16971> 2013-04-03 10:37:34.582447 7fcdffcdb700 0 log [INF] : pgmap v29017: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 23690B/s wr, 0op/s -16970> 2013-04-03 10:37:34.582449 7fcdffcdb700 10 send_log to self -16969> 2013-04-03 10:37:34.582450 7fcdffcdb700 10 log_queue is 2 last_log 83 sent 81 num 2 unsent 2 sending 2 -16968> 2013-04-03 10:37:34.582452 7fcdffcdb700 10 will send 2013-04-03 10:37:32.274687 mon.0 10.252.24.25:6789/0 82 : [INF] pgmap v29016: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -16967> 2013-04-03 10:37:34.582457 7fcdffcdb700 10 will send 2013-04-03 10:37:34.582448 mon.0 10.252.24.25:6789/0 83 : [INF] pgmap v29017: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 23690B/s wr, 0op/s -16966> 2013-04-03 10:37:34.582461 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x21fe800 -16965> 2013-04-03 10:37:34.582516 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos -16964> 2013-04-03 10:37:34.582536 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos version 394020 summary v 394020 -16963> 2013-04-03 10:37:34.582555 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -16962> 2013-04-03 10:37:34.582578 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x21fe800 con 0x1bac160 -16961> 2013-04-03 10:37:34.582601 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos -16960> 2013-04-03 10:37:34.582616 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos version 394020 summary v 394020 -16959> 2013-04-03 10:37:34.582626 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -16958> 2013-04-03 10:37:34.582637 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 preprocess_log log(2 entries) v1 from mon.0 -16957> 2013-04-03 10:37:34.582666 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -16956> 2013-04-03 10:37:34.582680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 prepare_log log(2 entries) v1 from mon.0 -16955> 2013-04-03 10:37:34.582691 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 logging 2013-04-03 10:37:32.274687 mon.0 10.252.24.25:6789/0 82 : [INF] pgmap v29016: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -16954> 2013-04-03 10:37:34.582711 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 logging 2013-04-03 10:37:34.582448 mon.0 10.252.24.25:6789/0 83 : [INF] pgmap v29017: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 23690B/s wr, 0op/s -16953> 2013-04-03 10:37:34.627514 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113295 ==== paxos(lease_ack lc 427320 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (2482775156 0 0) 0x2787c80 con 0x208a840 -16952> 2013-04-03 10:37:34.627562 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos -16951> 2013-04-03 10:37:34.627580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 update_from_paxos version 394020 summary v 394020 -16950> 2013-04-03 10:37:34.627596 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -16949> 2013-04-03 10:37:35.440593 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113296 ==== forward(pg_stats(0 pgs tid 59 v 605) v1) to leader v1 ==== 368+0+0 (3045548025 0 0) 0x29c5b80 con 0x208a840 -16948> 2013-04-03 10:37:35.440619 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 -16947> 2013-04-03 10:37:35.440630 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -16946> 2013-04-03 10:37:35.440681 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 preprocess_query pg_stats(0 pgs tid 59 v 605) v1 from osd.4 10.252.24.32:6800/13473 -16945> 2013-04-03 10:37:35.440689 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 prepare_update pg_stats(0 pgs tid 59 v 605) v1 from osd.4 10.252.24.32:6800/13473 -16944> 2013-04-03 10:37:35.440693 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 prepare_pg_stats pg_stats(0 pgs tid 59 v 605) v1 from osd.4 -16943> 2013-04-03 10:37:35.440699 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 got osd.4 osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[])) -16942> 2013-04-03 10:37:35.580959 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394020 encode_pending v394021 -16941> 2013-04-03 10:37:35.580990 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427320) queue_proposal bl 742 bytes; ctx = 0x1b91b80 -16940> 2013-04-03 10:37:35.580998 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427320) propose_queued 427321 742 bytes -16939> 2013-04-03 10:37:35.582175 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 60 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140347 v187) v2 ==== 289+0+0 (1759654578 0 0) 0x2a51000 con 0x2118420 -16938> 2013-04-03 10:37:35.582365 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427320 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4000 -16937> 2013-04-03 10:37:35.582450 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 encode_pending v 29018 -16936> 2013-04-03 10:37:35.582485 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427289..427320) queue_proposal bl 265 bytes; ctx = 0x1b91be0 -16935> 2013-04-03 10:37:35.582493 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427289..427320) propose_new_value not active; proposal queued -16934> 2013-04-03 10:37:35.582539 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140347 v187) v2 from mds.? 10.252.58.31:6800/10908 -16933> 2013-04-03 10:37:35.582558 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140347 v187) v2 -- ?+0 0x29c5b80 con 0x2118420 -16932> 2013-04-03 10:37:35.627543 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113297 ==== paxos(accept lc 427320 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (335915173 0 0) 0x2ab4000 con 0x208a840 -16931> 2013-04-03 10:37:35.752971 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427321 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787c80 -16930> 2013-04-03 10:37:35.752997 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427321 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2787500 -16929> 2013-04-03 10:37:35.753040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos -16928> 2013-04-03 10:37:35.753067 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos version 394021 summary v 394020 -16927> 2013-04-03 10:37:35.753082 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos latest full 393798 -16926> 2013-04-03 10:37:35.753100 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos applying incremental log 394021 2013-04-03 10:37:32.274687 mon.0 10.252.24.25:6789/0 82 : [INF] pgmap v29016: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -16925> 2013-04-03 10:37:35.753141 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos applying incremental log 394021 2013-04-03 10:37:34.582448 mon.0 10.252.24.25:6789/0 83 : [INF] pgmap v29017: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 23690B/s wr, 0op/s -16924> 2013-04-03 10:37:35.753183 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 check_subs -16923> 2013-04-03 10:37:35.753214 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 create_pending v 394022 -16922> 2013-04-03 10:37:35.753237 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 _updated_log for mon.0 10.252.24.25:6789/0 -16921> 2013-04-03 10:37:35.753250 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 83) v1 -- ?+0 0x2294e00 con 0x1bac160 -16920> 2013-04-03 10:37:35.753275 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427321) propose_queued 427322 265 bytes -16919> 2013-04-03 10:37:35.781067 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427321 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2a2ba00 -16918> 2013-04-03 10:37:35.781101 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 83) v1 ==== 0+0+0 (0 0 0) 0x2294e00 con 0x1bac160 -16917> 2013-04-03 10:37:35.781119 7fcdffcdb700 10 handle_log_ack log(last 83) v1 -16916> 2013-04-03 10:37:35.781122 7fcdffcdb700 10 logged 2013-04-03 10:37:32.274687 mon.0 10.252.24.25:6789/0 82 : [INF] pgmap v29016: 240 pgs: 169 active+clean, 71 stale+active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail -16915> 2013-04-03 10:37:35.781129 7fcdffcdb700 10 logged 2013-04-03 10:37:34.582448 mon.0 10.252.24.25:6789/0 83 : [INF] pgmap v29017: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5654 MB used, 1067 GB / 1079 GB avail; 23690B/s wr, 0op/s -16914> 2013-04-03 10:37:35.781135 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113298 ==== paxos(lease_ack lc 427321 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (3775042866 0 0) 0x2787500 con 0x208a840 -16913> 2013-04-03 10:37:35.836738 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113299 ==== paxos(accept lc 427321 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3172961336 0 0) 0x2a2ba00 con 0x208a840 -16912> 2013-04-03 10:37:35.837133 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427322 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787500 -16911> 2013-04-03 10:37:35.837162 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427322 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2ab4000 -16910> 2013-04-03 10:37:35.837247 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29017 update_from_paxos applying incremental 29018 -16909> 2013-04-03 10:37:35.837260 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 v29018: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 21882B/s wr, 0op/s -16908> 2013-04-03 10:37:35.837643 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 send_pg_creates to 0 pgs -16907> 2013-04-03 10:37:35.837648 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 update_logger -16906> 2013-04-03 10:37:35.837659 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 create_pending v 29019 -16905> 2013-04-03 10:37:35.837668 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 _updated_stats for osd.4 10.252.24.32:6800/13473 -16904> 2013-04-03 10:37:35.837675 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(0 pgs tid 59) v1 tid 17511) v2 -- ?+0 0x2b26d80 con 0x208a840 -16903> 2013-04-03 10:37:35.837685 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 check_osd_map already seen 605 >= 605 -16902> 2013-04-03 10:37:35.837688 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 update_logger -16901> 2013-04-03 10:37:35.837700 7fcdffcdb700 0 log [INF] : pgmap v29018: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 21882B/s wr, 0op/s -16900> 2013-04-03 10:37:35.837701 7fcdffcdb700 10 send_log to self -16899> 2013-04-03 10:37:35.837702 7fcdffcdb700 10 log_queue is 1 last_log 84 sent 83 num 1 unsent 1 sending 1 -16898> 2013-04-03 10:37:35.837704 7fcdffcdb700 10 will send 2013-04-03 10:37:35.837701 mon.0 10.252.24.25:6789/0 84 : [INF] pgmap v29018: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 21882B/s wr, 0op/s -16897> 2013-04-03 10:37:35.837710 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2545400 -16896> 2013-04-03 10:37:35.837763 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos -16895> 2013-04-03 10:37:35.837782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos version 394021 summary v 394021 -16894> 2013-04-03 10:37:35.837804 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -16893> 2013-04-03 10:37:35.837839 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2545400 con 0x1bac160 -16892> 2013-04-03 10:37:35.837863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos -16891> 2013-04-03 10:37:35.837879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos version 394021 summary v 394021 -16890> 2013-04-03 10:37:35.837888 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16889> 2013-04-03 10:37:35.837899 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 preprocess_log log(1 entries) v1 from mon.0 -16888> 2013-04-03 10:37:35.837922 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16887> 2013-04-03 10:37:35.837933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 prepare_log log(1 entries) v1 from mon.0 -16886> 2013-04-03 10:37:35.837943 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 logging 2013-04-03 10:37:35.837701 mon.0 10.252.24.25:6789/0 84 : [INF] pgmap v29018: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 21882B/s wr, 0op/s -16885> 2013-04-03 10:37:35.850400 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113300 ==== paxos(lease_ack lc 427322 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (1240054812 0 0) 0x2ab4000 con 0x208a840 -16884> 2013-04-03 10:37:35.850439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos -16883> 2013-04-03 10:37:35.850456 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 update_from_paxos version 394021 summary v 394021 -16882> 2013-04-03 10:37:35.850473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -16881> 2013-04-03 10:37:36.579253 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113301 ==== forward(pg_stats(45 pgs tid 21 v 605) v1) to leader v1 ==== 17165+0+0 (2750896767 0 0) 0x29c4840 con 0x208a840 -16880> 2013-04-03 10:37:36.579275 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -16879> 2013-04-03 10:37:36.579285 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26d80 from 10.252.24.88:6789/0 -16878> 2013-04-03 10:37:36.579337 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 preprocess_query pg_stats(45 pgs tid 21 v 605) v1 from osd.1 10.252.24.25:6805/3572 -16877> 2013-04-03 10:37:36.579346 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 prepare_update pg_stats(45 pgs tid 21 v 605) v1 from osd.1 10.252.24.25:6805/3572 -16876> 2013-04-03 10:37:36.579350 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 prepare_pg_stats pg_stats(45 pgs tid 21 v 605) v1 from osd.1 -16875> 2013-04-03 10:37:36.579357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,2,3,4]/[])) -16874> 2013-04-03 10:37:36.744815 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113302 ==== forward(pg_stats(73 pgs tid 30441 v 605) v1) to leader v1 ==== 27613+0+0 (779223415 0 0) 0x29c5080 con 0x208a840 -16873> 2013-04-03 10:37:36.744832 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -16872> 2013-04-03 10:37:36.744841 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -16871> 2013-04-03 10:37:36.744870 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 preprocess_query pg_stats(73 pgs tid 30441 v 605) v1 from osd.3 10.252.24.88:6801/17538 -16870> 2013-04-03 10:37:36.744878 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 prepare_update pg_stats(73 pgs tid 30441 v 605) v1 from osd.3 10.252.24.88:6801/17538 -16869> 2013-04-03 10:37:36.744882 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 prepare_pg_stats pg_stats(73 pgs tid 30441 v 605) v1 from osd.3 -16868> 2013-04-03 10:37:36.744886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 got osd.3 osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -16867> 2013-04-03 10:37:36.809644 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113303 ==== forward(pg_stats(51 pgs tid 408 v 605) v1) to leader v1 ==== 19407+0+0 (3566209425 0 0) 0x29c42c0 con 0x208a840 -16866> 2013-04-03 10:37:36.809660 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -16865> 2013-04-03 10:37:36.809668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -16864> 2013-04-03 10:37:36.809697 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 preprocess_query pg_stats(51 pgs tid 408 v 605) v1 from osd.0 10.252.58.31:6802/2682 -16863> 2013-04-03 10:37:36.809704 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 prepare_update pg_stats(51 pgs tid 408 v 605) v1 from osd.0 10.252.58.31:6802/2682 -16862> 2013-04-03 10:37:36.809708 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 prepare_pg_stats pg_stats(51 pgs tid 408 v 605) v1 from osd.0 -16861> 2013-04-03 10:37:36.809713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 got osd.0 osd_stat(1139 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1139 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -16860> 2013-04-03 10:37:36.836941 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394021 encode_pending v394022 -16859> 2013-04-03 10:37:36.836977 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427322) queue_proposal bl 455 bytes; ctx = 0x1b91bc0 -16858> 2013-04-03 10:37:36.836986 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427322) propose_queued 427323 455 bytes -16857> 2013-04-03 10:37:36.838500 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427322 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4280 -16856> 2013-04-03 10:37:36.838589 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 encode_pending v 29019 -16855> 2013-04-03 10:37:36.839025 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427289..427322) queue_proposal bl 63446 bytes; ctx = 0x1b91b40 -16854> 2013-04-03 10:37:36.839037 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427289..427322) propose_new_value not active; proposal queued -16853> 2013-04-03 10:37:36.875503 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113304 ==== paxos(accept lc 427322 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (375545055 0 0) 0x2ab4280 con 0x208a840 -16852> 2013-04-03 10:37:36.876139 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427323 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4000 -16851> 2013-04-03 10:37:36.876162 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427323 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2a2ba00 -16850> 2013-04-03 10:37:36.876205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos -16849> 2013-04-03 10:37:36.876243 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos version 394022 summary v 394021 -16848> 2013-04-03 10:37:36.876258 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos latest full 393798 -16847> 2013-04-03 10:37:36.876280 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos applying incremental log 394022 2013-04-03 10:37:35.837701 mon.0 10.252.24.25:6789/0 84 : [INF] pgmap v29018: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 21882B/s wr, 0op/s -16846> 2013-04-03 10:37:36.876328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 check_subs -16845> 2013-04-03 10:37:36.876361 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 create_pending v 394023 -16844> 2013-04-03 10:37:36.876384 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 _updated_log for mon.0 10.252.24.25:6789/0 -16843> 2013-04-03 10:37:36.876398 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 84) v1 -- ?+0 0x28a5380 con 0x1bac160 -16842> 2013-04-03 10:37:36.876423 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427323) propose_queued 427324 63446 bytes -16841> 2013-04-03 10:37:36.882704 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427323 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2a2bc80 -16840> 2013-04-03 10:37:36.882747 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 84) v1 ==== 0+0+0 (0 0 0) 0x28a5380 con 0x1bac160 -16839> 2013-04-03 10:37:36.882756 7fcdffcdb700 10 handle_log_ack log(last 84) v1 -16838> 2013-04-03 10:37:36.882758 7fcdffcdb700 10 logged 2013-04-03 10:37:35.837701 mon.0 10.252.24.25:6789/0 84 : [INF] pgmap v29018: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 21882B/s wr, 0op/s -16837> 2013-04-03 10:37:36.901600 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113305 ==== paxos(lease_ack lc 427323 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (242305879 0 0) 0x2a2bc80 con 0x208a840 -16836> 2013-04-03 10:37:36.927563 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113306 ==== paxos(accept lc 427323 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2917222530 0 0) 0x2a2ba00 con 0x208a840 -16835> 2013-04-03 10:37:36.928934 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427324 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2a2bc80 -16834> 2013-04-03 10:37:36.928954 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427324 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2ab4280 -16833> 2013-04-03 10:37:36.929081 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29018 update_from_paxos applying incremental 29019 -16832> 2013-04-03 10:37:36.929496 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 v29019: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail -16831> 2013-04-03 10:37:36.929537 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 send_pg_creates to 0 pgs -16830> 2013-04-03 10:37:36.929544 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 update_logger -16829> 2013-04-03 10:37:36.929587 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 create_pending v 29020 -16828> 2013-04-03 10:37:36.929601 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 _updated_stats for osd.1 10.252.24.25:6805/3572 -16827> 2013-04-03 10:37:36.929610 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(45 pgs tid 21) v1 tid 17512) v2 -- ?+0 0x24f8d80 con 0x208a840 -16826> 2013-04-03 10:37:36.929628 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 _updated_stats for osd.3 10.252.24.88:6801/17538 -16825> 2013-04-03 10:37:36.929634 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(73 pgs tid 30441) v1 tid 17513) v2 -- ?+0 0x2b26d80 con 0x208a840 -16824> 2013-04-03 10:37:36.929688 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 _updated_stats for osd.0 10.252.58.31:6802/2682 -16823> 2013-04-03 10:37:36.929698 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(51 pgs tid 408) v1 tid 17514) v2 -- ?+0 0x24f8900 con 0x208a840 -16822> 2013-04-03 10:37:36.929732 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 check_osd_map already seen 605 >= 605 -16821> 2013-04-03 10:37:36.929741 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 update_logger -16820> 2013-04-03 10:37:36.929754 7fcdffcdb700 0 log [INF] : pgmap v29019: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail -16819> 2013-04-03 10:37:36.929756 7fcdffcdb700 10 send_log to self -16818> 2013-04-03 10:37:36.929757 7fcdffcdb700 10 log_queue is 1 last_log 85 sent 84 num 1 unsent 1 sending 1 -16817> 2013-04-03 10:37:36.929758 7fcdffcdb700 10 will send 2013-04-03 10:37:36.929756 mon.0 10.252.24.25:6789/0 85 : [INF] pgmap v29019: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail -16816> 2013-04-03 10:37:36.929765 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287ec00 -16815> 2013-04-03 10:37:36.929835 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos -16814> 2013-04-03 10:37:36.929857 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos version 394022 summary v 394022 -16813> 2013-04-03 10:37:36.929876 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -16812> 2013-04-03 10:37:36.929897 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287ec00 con 0x1bac160 -16811> 2013-04-03 10:37:36.929917 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos -16810> 2013-04-03 10:37:36.929933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos version 394022 summary v 394022 -16809> 2013-04-03 10:37:36.929944 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16808> 2013-04-03 10:37:36.929955 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 preprocess_log log(1 entries) v1 from mon.0 -16807> 2013-04-03 10:37:36.929978 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16806> 2013-04-03 10:37:36.929990 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 prepare_log log(1 entries) v1 from mon.0 -16805> 2013-04-03 10:37:36.930000 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 logging 2013-04-03 10:37:36.929756 mon.0 10.252.24.25:6789/0 85 : [INF] pgmap v29019: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail -16804> 2013-04-03 10:37:36.960728 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113307 ==== paxos(lease_ack lc 427324 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (1073557262 0 0) 0x2ab4280 con 0x208a840 -16803> 2013-04-03 10:37:36.960774 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos -16802> 2013-04-03 10:37:36.960793 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 update_from_paxos version 394022 summary v 394022 -16801> 2013-04-03 10:37:36.960809 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -16800> 2013-04-03 10:37:36.960826 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113308 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 62 v187) v2) to leader v1 ==== 560+0+0 (4214606947 0 0) 0x29c4580 con 0x208a840 -16799> 2013-04-03 10:37:36.960835 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -16798> 2013-04-03 10:37:36.960844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29c4b00 from 10.252.24.88:6789/0 -16797> 2013-04-03 10:37:36.960868 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 62 v187) v2 from mds.0 10.252.24.25:6802/3144 -16796> 2013-04-03 10:37:36.960879 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 62 v187) v2 tid 17515) v2 -- ?+0 0x24f8b40 con 0x208a840 -16795> 2013-04-03 10:37:37.381763 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 60 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21718 v187) v2 ==== 289+0+0 (2494308771 0 0) 0x2a84340 con 0x200d420 -16794> 2013-04-03 10:37:37.381838 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21718 v187) v2 from mds.? 10.252.24.88:6800/17213 -16793> 2013-04-03 10:37:37.381852 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21718 v187) v2 -- ?+0 0x29c4580 con 0x200d420 -16792> 2013-04-03 10:37:37.927768 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394022 encode_pending v394023 -16791> 2013-04-03 10:37:37.927797 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427324) queue_proposal bl 435 bytes; ctx = 0x1b91b30 -16790> 2013-04-03 10:37:37.927807 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427324) propose_queued 427325 435 bytes -16789> 2013-04-03 10:37:37.929364 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427324 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4780 -16788> 2013-04-03 10:37:37.952547 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113309 ==== paxos(accept lc 427324 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (133688854 0 0) 0x2ab4780 con 0x208a840 -16787> 2013-04-03 10:37:37.953473 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427325 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4280 -16786> 2013-04-03 10:37:37.953509 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427325 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2a2ba00 -16785> 2013-04-03 10:37:37.953542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos -16784> 2013-04-03 10:37:37.953568 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos version 394023 summary v 394022 -16783> 2013-04-03 10:37:37.953582 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos latest full 393798 -16782> 2013-04-03 10:37:37.953599 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos applying incremental log 394023 2013-04-03 10:37:36.929756 mon.0 10.252.24.25:6789/0 85 : [INF] pgmap v29019: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail -16781> 2013-04-03 10:37:37.953648 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 check_subs -16780> 2013-04-03 10:37:37.953679 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 create_pending v 394024 -16779> 2013-04-03 10:37:37.953702 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 _updated_log for mon.0 10.252.24.25:6789/0 -16778> 2013-04-03 10:37:37.953715 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 85) v1 -- ?+0 0x28a5380 con 0x1bac160 -16777> 2013-04-03 10:37:37.954044 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos -16776> 2013-04-03 10:37:37.954066 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos version 394023 summary v 394023 -16775> 2013-04-03 10:37:37.954086 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -16774> 2013-04-03 10:37:37.954107 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 85) v1 ==== 0+0+0 (0 0 0) 0x28a5380 con 0x1bac160 -16773> 2013-04-03 10:37:37.954115 7fcdffcdb700 10 handle_log_ack log(last 85) v1 -16772> 2013-04-03 10:37:37.954116 7fcdffcdb700 10 logged 2013-04-03 10:37:36.929756 mon.0 10.252.24.25:6789/0 85 : [INF] pgmap v29019: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail -16771> 2013-04-03 10:37:37.978516 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113310 ==== paxos(lease_ack lc 427325 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (2264101555 0 0) 0x2a2ba00 con 0x208a840 -16770> 2013-04-03 10:37:37.978560 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos -16769> 2013-04-03 10:37:37.978577 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos version 394023 summary v 394023 -16768> 2013-04-03 10:37:37.978592 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -16767> 2013-04-03 10:37:38.217760 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113311 ==== forward(pg_stats(65 pgs tid 2 v 605) v1) to leader v1 ==== 24629+0+0 (3204416483 0 0) 0x29c42c0 con 0x208a840 -16766> 2013-04-03 10:37:38.217787 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -16765> 2013-04-03 10:37:38.217798 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -16764> 2013-04-03 10:37:38.217848 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 preprocess_query pg_stats(65 pgs tid 2 v 605) v1 from osd.2 10.252.24.30:6801/5656 -16763> 2013-04-03 10:37:38.217856 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 prepare_update pg_stats(65 pgs tid 2 v 605) v1 from osd.2 10.252.24.30:6801/5656 -16762> 2013-04-03 10:37:38.217860 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 prepare_pg_stats pg_stats(65 pgs tid 2 v 605) v1 from osd.2 -16761> 2013-04-03 10:37:38.217867 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,3]/[])) -16760> 2013-04-03 10:37:38.336554 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 check_down_pgs -16759> 2013-04-03 10:37:38.336585 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 v29019: 240 pgs: 65 active, 175 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail -16758> 2013-04-03 10:37:38.336613 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -16757> 2013-04-03 10:37:38.336638 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -16756> 2013-04-03 10:37:38.336675 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -16755> 2013-04-03 10:37:38.336706 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos -16754> 2013-04-03 10:37:38.336726 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos version 394023 summary v 394023 -16753> 2013-04-03 10:37:38.336735 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 log -16752> 2013-04-03 10:37:38.336752 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -16751> 2013-04-03 10:37:38.336771 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -16750> 2013-04-03 10:37:38.952746 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 encode_pending v 29020 -16749> 2013-04-03 10:37:38.952939 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427289..427325) queue_proposal bl 24526 bytes; ctx = 0x1b91ba0 -16748> 2013-04-03 10:37:38.952949 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427289..427325) propose_queued 427326 24526 bytes -16747> 2013-04-03 10:37:38.955104 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427325 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4500 -16746> 2013-04-03 10:37:38.986934 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113312 ==== paxos(accept lc 427325 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3088661737 0 0) 0x2ab4500 con 0x208a840 -16745> 2013-04-03 10:37:38.987973 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427326 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2a2ba00 -16744> 2013-04-03 10:37:38.987991 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427326 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2ab4780 -16743> 2013-04-03 10:37:38.988081 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29019 update_from_paxos applying incremental 29020 -16742> 2013-04-03 10:37:38.988249 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29020 v29020: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail -16741> 2013-04-03 10:37:38.988278 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29020 send_pg_creates to 0 pgs -16740> 2013-04-03 10:37:38.988282 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29020 update_logger -16739> 2013-04-03 10:37:38.988299 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29020 create_pending v 29021 -16738> 2013-04-03 10:37:38.988309 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29020 _updated_stats for osd.2 10.252.24.30:6801/5656 -16737> 2013-04-03 10:37:38.988317 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(65 pgs tid 2) v1 tid 17516) v2 -- ?+0 0x24f8900 con 0x208a840 -16736> 2013-04-03 10:37:38.988343 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29020 check_osd_map already seen 605 >= 605 -16735> 2013-04-03 10:37:38.988347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29020 update_logger -16734> 2013-04-03 10:37:38.988362 7fcdffcdb700 0 log [INF] : pgmap v29020: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail -16733> 2013-04-03 10:37:38.988364 7fcdffcdb700 10 send_log to self -16732> 2013-04-03 10:37:38.988365 7fcdffcdb700 10 log_queue is 1 last_log 86 sent 85 num 1 unsent 1 sending 1 -16731> 2013-04-03 10:37:38.988366 7fcdffcdb700 10 will send 2013-04-03 10:37:38.988363 mon.0 10.252.24.25:6789/0 86 : [INF] pgmap v29020: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail -16730> 2013-04-03 10:37:38.988373 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287ec00 -16729> 2013-04-03 10:37:38.988425 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos -16728> 2013-04-03 10:37:38.988446 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos version 394023 summary v 394023 -16727> 2013-04-03 10:37:38.988465 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -16726> 2013-04-03 10:37:38.988486 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287ec00 con 0x1bac160 -16725> 2013-04-03 10:37:38.988508 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos -16724> 2013-04-03 10:37:38.988523 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos version 394023 summary v 394023 -16723> 2013-04-03 10:37:38.988533 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16722> 2013-04-03 10:37:38.988544 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 preprocess_log log(1 entries) v1 from mon.0 -16721> 2013-04-03 10:37:38.988567 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16720> 2013-04-03 10:37:38.988579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 prepare_log log(1 entries) v1 from mon.0 -16719> 2013-04-03 10:37:38.988589 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 logging 2013-04-03 10:37:38.988363 mon.0 10.252.24.25:6789/0 86 : [INF] pgmap v29020: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail -16718> 2013-04-03 10:37:39.021362 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113313 ==== paxos(lease_ack lc 427326 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (1625908939 0 0) 0x2ab4780 con 0x208a840 -16717> 2013-04-03 10:37:39.021405 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos -16716> 2013-04-03 10:37:39.021422 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394023 update_from_paxos version 394023 summary v 394023 -16715> 2013-04-03 10:37:39.021442 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -16714> 2013-04-03 10:37:39.193215 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) -16713> 2013-04-03 10:37:39.193259 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader -16712> 2013-04-03 10:37:39.582322 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 61 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140348 v187) v2 ==== 289+0+0 (2162888385 0 0) 0x2a51dc0 con 0x2118420 -16711> 2013-04-03 10:37:39.582389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140348 v187) v2 from mds.? 10.252.58.31:6800/10908 -16710> 2013-04-03 10:37:39.582401 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140348 v187) v2 -- ?+0 0x29c42c0 con 0x2118420 -16709> 2013-04-03 10:37:39.987141 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394023 encode_pending v394024 -16708> 2013-04-03 10:37:39.987170 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427289..427326) queue_proposal bl 424 bytes; ctx = 0x1b91bb0 -16707> 2013-04-03 10:37:39.987180 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427289..427326) propose_queued 427327 424 bytes -16706> 2013-04-03 10:37:39.988272 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427326 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4c80 -16705> 2013-04-03 10:37:40.014134 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113314 ==== paxos(accept lc 427326 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2584875115 0 0) 0x2ab4c80 con 0x208a840 -16704> 2013-04-03 10:37:40.014938 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427327 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4780 -16703> 2013-04-03 10:37:40.014964 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427327 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2ab4500 -16702> 2013-04-03 10:37:40.015002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos -16701> 2013-04-03 10:37:40.015026 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394023 -16700> 2013-04-03 10:37:40.015040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos latest full 393798 -16699> 2013-04-03 10:37:40.015056 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394024 update_from_paxos applying incremental log 394024 2013-04-03 10:37:38.988363 mon.0 10.252.24.25:6789/0 86 : [INF] pgmap v29020: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail -16698> 2013-04-03 10:37:40.015102 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 check_subs -16697> 2013-04-03 10:37:40.015133 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 create_pending v 394025 -16696> 2013-04-03 10:37:40.015155 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394024 _updated_log for mon.0 10.252.24.25:6789/0 -16695> 2013-04-03 10:37:40.015167 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 86) v1 -- ?+0 0x28a5700 con 0x1bac160 -16694> 2013-04-03 10:37:40.015218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos -16693> 2013-04-03 10:37:40.015234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394024 -16692> 2013-04-03 10:37:40.015251 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16691> 2013-04-03 10:37:40.015273 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 86) v1 ==== 0+0+0 (0 0 0) 0x28a5700 con 0x1bac160 -16690> 2013-04-03 10:37:40.015283 7fcdffcdb700 10 handle_log_ack log(last 86) v1 -16689> 2013-04-03 10:37:40.015286 7fcdffcdb700 10 logged 2013-04-03 10:37:38.988363 mon.0 10.252.24.25:6789/0 86 : [INF] pgmap v29020: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail -16688> 2013-04-03 10:37:40.040253 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113315 ==== paxos(lease_ack lc 427327 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (1773555212 0 0) 0x2ab4500 con 0x208a840 -16687> 2013-04-03 10:37:40.040297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos -16686> 2013-04-03 10:37:40.040314 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394024 -16685> 2013-04-03 10:37:40.040328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16684> 2013-04-03 10:37:40.928361 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113316 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 63 v187) v2) to leader v1 ==== 560+0+0 (335387407 0 0) 0x29c5340 con 0x208a840 -16683> 2013-04-03 10:37:40.928382 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -16682> 2013-04-03 10:37:40.928391 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c5600 from 10.252.24.88:6789/0 -16681> 2013-04-03 10:37:40.928443 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 63 v187) v2 from mds.0 10.252.24.25:6802/3144 -16680> 2013-04-03 10:37:40.928454 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 63 v187) v2 tid 17517) v2 -- ?+0 0x24f8b40 con 0x208a840 -16679> 2013-04-03 10:37:41.381884 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 61 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21719 v187) v2 ==== 289+0+0 (3082985182 0 0) 0x2a84b80 con 0x200d420 -16678> 2013-04-03 10:37:41.381953 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21719 v187) v2 from mds.? 10.252.24.88:6800/17213 -16677> 2013-04-03 10:37:41.381966 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21719 v187) v2 -- ?+0 0x29c5340 con 0x200d420 -16676> 2013-04-03 10:37:43.015041 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427327 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29b4000 -16675> 2013-04-03 10:37:43.015831 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113317 ==== paxos(lease_ack lc 427327 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (1172100926 0 0) 0x29b4000 con 0x208a840 -16674> 2013-04-03 10:37:43.015903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos -16673> 2013-04-03 10:37:43.015923 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394024 -16672> 2013-04-03 10:37:43.015940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16671> 2013-04-03 10:37:43.213804 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113318 ==== forward(pg_stats(1 pgs tid 3 v 605) v1) to leader v1 ==== 757+0+0 (3864942509 0 0) 0x2a51dc0 con 0x208a840 -16670> 2013-04-03 10:37:43.213833 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -16669> 2013-04-03 10:37:43.213843 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -16668> 2013-04-03 10:37:43.213894 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29020 preprocess_query pg_stats(1 pgs tid 3 v 605) v1 from osd.2 10.252.24.30:6801/5656 -16667> 2013-04-03 10:37:43.213901 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29020 prepare_update pg_stats(1 pgs tid 3 v 605) v1 from osd.2 10.252.24.30:6801/5656 -16666> 2013-04-03 10:37:43.213905 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29020 prepare_pg_stats pg_stats(1 pgs tid 3 v 605) v1 from osd.2 -16665> 2013-04-03 10:37:43.213911 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29020 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -16664> 2013-04-03 10:37:43.264290 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29020 encode_pending v 29021 -16663> 2013-04-03 10:37:43.264427 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427289..427327) queue_proposal bl 654 bytes; ctx = 0x1b91b50 -16662> 2013-04-03 10:37:43.264439 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427289..427327) propose_queued 427328 654 bytes -16661> 2013-04-03 10:37:43.265938 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427327 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4280 -16660> 2013-04-03 10:37:43.336890 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -16659> 2013-04-03 10:37:43.336933 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -16658> 2013-04-03 10:37:43.336952 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos -16657> 2013-04-03 10:37:43.336971 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394024 -16656> 2013-04-03 10:37:43.336980 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394024 log -16655> 2013-04-03 10:37:43.336997 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16654> 2013-04-03 10:37:43.337015 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -16653> 2013-04-03 10:37:43.389258 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113319 ==== paxos(accept lc 427327 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3565628304 0 0) 0x29b4280 con 0x208a840 -16652> 2013-04-03 10:37:43.389560 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427328 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4000 -16651> 2013-04-03 10:37:43.389591 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427328 fc 427289 pn 0 opn 0) v3 -- ?+0 0x2ab4500 -16650> 2013-04-03 10:37:43.389660 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29020 update_from_paxos applying incremental 29021 -16649> 2013-04-03 10:37:43.389675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 v29021: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2549B/s wr, 0op/s -16648> 2013-04-03 10:37:43.389693 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 send_pg_creates to 0 pgs -16647> 2013-04-03 10:37:43.389695 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 update_logger -16646> 2013-04-03 10:37:43.389705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 create_pending v 29022 -16645> 2013-04-03 10:37:43.389713 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29021 _updated_stats for osd.2 10.252.24.30:6801/5656 -16644> 2013-04-03 10:37:43.389719 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 3) v1 tid 17518) v2 -- ?+0 0x24f8900 con 0x208a840 -16643> 2013-04-03 10:37:43.389729 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 check_osd_map already seen 605 >= 605 -16642> 2013-04-03 10:37:43.389732 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 update_logger -16641> 2013-04-03 10:37:43.389742 7fcdffcdb700 0 log [INF] : pgmap v29021: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2549B/s wr, 0op/s -16640> 2013-04-03 10:37:43.389748 7fcdffcdb700 10 send_log to self -16639> 2013-04-03 10:37:43.389749 7fcdffcdb700 10 log_queue is 1 last_log 87 sent 86 num 1 unsent 1 sending 1 -16638> 2013-04-03 10:37:43.389751 7fcdffcdb700 10 will send 2013-04-03 10:37:43.389748 mon.0 10.252.24.25:6789/0 87 : [INF] pgmap v29021: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2549B/s wr, 0op/s -16637> 2013-04-03 10:37:43.389757 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287ec00 -16636> 2013-04-03 10:37:43.389802 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos -16635> 2013-04-03 10:37:43.389827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394024 -16634> 2013-04-03 10:37:43.389845 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16633> 2013-04-03 10:37:43.389865 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287ec00 con 0x1bac160 -16632> 2013-04-03 10:37:43.389887 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos -16631> 2013-04-03 10:37:43.389902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394024 -16630> 2013-04-03 10:37:43.389909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16629> 2013-04-03 10:37:43.389919 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 preprocess_log log(1 entries) v1 from mon.0 -16628> 2013-04-03 10:37:43.389941 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16627> 2013-04-03 10:37:43.389953 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 prepare_log log(1 entries) v1 from mon.0 -16626> 2013-04-03 10:37:43.389964 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 logging 2013-04-03 10:37:43.389748 mon.0 10.252.24.25:6789/0 87 : [INF] pgmap v29021: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2549B/s wr, 0op/s -16625> 2013-04-03 10:37:43.402439 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113320 ==== paxos(lease_ack lc 427328 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (3835940507 0 0) 0x2ab4500 con 0x208a840 -16624> 2013-04-03 10:37:43.402474 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos -16623> 2013-04-03 10:37:43.402489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394024 update_from_paxos version 394024 summary v 394024 -16622> 2013-04-03 10:37:43.402503 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16621> 2013-04-03 10:37:43.582428 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 62 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140349 v187) v2 ==== 289+0+0 (2743447996 0 0) 0x2a52080 con 0x2118420 -16620> 2013-04-03 10:37:43.582476 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140349 v187) v2 from mds.? 10.252.58.31:6800/10908 -16619> 2013-04-03 10:37:43.582486 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140349 v187) v2 -- ?+0 0x2a51dc0 con 0x2118420 -16618> 2013-04-03 10:37:44.389462 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394024 encode_pending v394025 -16617> 2013-04-03 10:37:44.389489 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427289..427328) queue_proposal bl 443 bytes; ctx = 0x1b91fe0 -16616> 2013-04-03 10:37:44.389497 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427289..427328) propose_queued 427329 443 bytes -16615> 2013-04-03 10:37:44.391381 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427328 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dcc80 -16614> 2013-04-03 10:37:44.417024 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113321 ==== paxos(accept lc 427328 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2400408705 0 0) 0x26dcc80 con 0x208a840 -16613> 2013-04-03 10:37:44.417343 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427329 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4500 -16612> 2013-04-03 10:37:44.417360 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427329 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29b4280 -16611> 2013-04-03 10:37:44.417394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos -16610> 2013-04-03 10:37:44.417420 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394024 -16609> 2013-04-03 10:37:44.417434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos latest full 393798 -16608> 2013-04-03 10:37:44.417451 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394025 update_from_paxos applying incremental log 394025 2013-04-03 10:37:43.389748 mon.0 10.252.24.25:6789/0 87 : [INF] pgmap v29021: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2549B/s wr, 0op/s -16607> 2013-04-03 10:37:44.417496 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 check_subs -16606> 2013-04-03 10:37:44.417527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 create_pending v 394026 -16605> 2013-04-03 10:37:44.417681 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394025 _updated_log for mon.0 10.252.24.25:6789/0 -16604> 2013-04-03 10:37:44.417699 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 87) v1 -- ?+0 0x28a58c0 con 0x1bac160 -16603> 2013-04-03 10:37:44.417751 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos -16602> 2013-04-03 10:37:44.417767 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 -16601> 2013-04-03 10:37:44.417783 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16600> 2013-04-03 10:37:44.417805 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 87) v1 ==== 0+0+0 (0 0 0) 0x28a58c0 con 0x1bac160 -16599> 2013-04-03 10:37:44.417820 7fcdffcdb700 10 handle_log_ack log(last 87) v1 -16598> 2013-04-03 10:37:44.417824 7fcdffcdb700 10 logged 2013-04-03 10:37:43.389748 mon.0 10.252.24.25:6789/0 87 : [INF] pgmap v29021: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2549B/s wr, 0op/s -16597> 2013-04-03 10:37:44.430262 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113322 ==== paxos(lease_ack lc 427329 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (1131911829 0 0) 0x29b4280 con 0x208a840 -16596> 2013-04-03 10:37:44.430297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos -16595> 2013-04-03 10:37:44.430314 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 -16594> 2013-04-03 10:37:44.430328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16593> 2013-04-03 10:37:44.928573 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113323 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 64 v187) v2) to leader v1 ==== 560+0+0 (3211453114 0 0) 0x29c4000 con 0x208a840 -16592> 2013-04-03 10:37:44.928594 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -16591> 2013-04-03 10:37:44.928604 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2145000 from 10.252.24.88:6789/0 -16590> 2013-04-03 10:37:44.928655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 64 v187) v2 from mds.0 10.252.24.25:6802/3144 -16589> 2013-04-03 10:37:44.928666 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 64 v187) v2 tid 17519) v2 -- ?+0 0x24f8b40 con 0x208a840 -16588> 2013-04-03 10:37:45.381972 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 62 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21720 v187) v2 ==== 289+0+0 (1095905204 0 0) 0x2a848c0 con 0x200d420 -16587> 2013-04-03 10:37:45.382048 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21720 v187) v2 from mds.? 10.252.24.88:6800/17213 -16586> 2013-04-03 10:37:45.382061 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21720 v187) v2 -- ?+0 0x29c4000 con 0x200d420 -16585> 2013-04-03 10:37:47.417434 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427329 fc 427289 pn 0 opn 0) v3 -- ?+0 0x26dc280 -16584> 2013-04-03 10:37:47.418179 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113324 ==== paxos(lease_ack lc 427329 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (2392517320 0 0) 0x26dc280 con 0x208a840 -16583> 2013-04-03 10:37:47.418253 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos -16582> 2013-04-03 10:37:47.418272 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 -16581> 2013-04-03 10:37:47.418289 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16580> 2013-04-03 10:37:47.582549 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 63 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140350 v187) v2 ==== 289+0+0 (3325456443 0 0) 0x2a528c0 con 0x2118420 -16579> 2013-04-03 10:37:47.582596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140350 v187) v2 from mds.? 10.252.58.31:6800/10908 -16578> 2013-04-03 10:37:47.582609 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140350 v187) v2 -- ?+0 0x2a848c0 con 0x2118420 -16577> 2013-04-03 10:37:48.337162 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29021 check_down_pgs -16576> 2013-04-03 10:37:48.337192 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29021 v29021: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2549B/s wr, 0op/s -16575> 2013-04-03 10:37:48.337219 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -16574> 2013-04-03 10:37:48.337248 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -16573> 2013-04-03 10:37:48.337285 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -16572> 2013-04-03 10:37:48.337324 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos -16571> 2013-04-03 10:37:48.337344 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 -16570> 2013-04-03 10:37:48.337353 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394025 log -16569> 2013-04-03 10:37:48.337370 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16568> 2013-04-03 10:37:48.337389 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -16567> 2013-04-03 10:37:48.928649 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113325 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 65 v187) v2) to leader v1 ==== 560+0+0 (2788757618 0 0) 0x2a52080 con 0x208a840 -16566> 2013-04-03 10:37:48.928667 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -16565> 2013-04-03 10:37:48.928676 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x295d580 from 10.252.24.88:6789/0 -16564> 2013-04-03 10:37:48.928714 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 65 v187) v2 from mds.0 10.252.24.25:6802/3144 -16563> 2013-04-03 10:37:48.928725 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 65 v187) v2 tid 17520) v2 -- ?+0 0x24f8900 con 0x208a840 -16562> 2013-04-03 10:37:49.382099 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 63 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21721 v187) v2 ==== 289+0+0 (1648136393 0 0) 0x2995000 con 0x200d420 -16561> 2013-04-03 10:37:49.382179 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21721 v187) v2 from mds.? 10.252.24.88:6800/17213 -16560> 2013-04-03 10:37:49.382192 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21721 v187) v2 -- ?+0 0x2a52080 con 0x200d420 -16559> 2013-04-03 10:37:50.417506 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427329 fc 427289 pn 0 opn 0) v3 -- ?+0 0x20f4c80 -16558> 2013-04-03 10:37:50.418305 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113326 ==== paxos(lease_ack lc 427329 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (757869132 0 0) 0x20f4c80 con 0x208a840 -16557> 2013-04-03 10:37:50.418379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos -16556> 2013-04-03 10:37:50.418399 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 -16555> 2013-04-03 10:37:50.418415 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16554> 2013-04-03 10:37:51.577452 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113327 ==== forward(pg_stats(1 pgs tid 22 v 605) v1) to leader v1 ==== 753+0+0 (606043240 0 0) 0x2a528c0 con 0x208a840 -16553> 2013-04-03 10:37:51.577473 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -16552> 2013-04-03 10:37:51.577483 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -16551> 2013-04-03 10:37:51.577533 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 preprocess_query pg_stats(1 pgs tid 22 v 605) v1 from osd.1 10.252.24.25:6805/3572 -16550> 2013-04-03 10:37:51.577541 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 prepare_update pg_stats(1 pgs tid 22 v 605) v1 from osd.1 10.252.24.25:6805/3572 -16549> 2013-04-03 10:37:51.577545 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 prepare_pg_stats pg_stats(1 pgs tid 22 v 605) v1 from osd.1 -16548> 2013-04-03 10:37:51.577553 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29021 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -16547> 2013-04-03 10:37:51.582663 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 64 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140351 v187) v2 ==== 289+0+0 (3847802694 0 0) 0x2a832c0 con 0x2118420 -16546> 2013-04-03 10:37:51.582708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140351 v187) v2 from mds.? 10.252.58.31:6800/10908 -16545> 2013-04-03 10:37:51.582718 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140351 v187) v2 -- ?+0 0x2a528c0 con 0x2118420 -16544> 2013-04-03 10:37:51.627736 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29021 encode_pending v 29022 -16543> 2013-04-03 10:37:51.627772 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427289..427329) queue_proposal bl 650 bytes; ctx = 0x1b91ed0 -16542> 2013-04-03 10:37:51.627781 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427289..427329) propose_queued 427330 650 bytes -16541> 2013-04-03 10:37:51.628895 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427329 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0780 -16540> 2013-04-03 10:37:51.657024 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113328 ==== paxos(accept lc 427329 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1027629910 0 0) 0x29e0780 con 0x208a840 -16539> 2013-04-03 10:37:51.657656 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427330 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4c80 -16538> 2013-04-03 10:37:51.657687 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427330 fc 427289 pn 0 opn 0) v3 -- ?+0 0x26dc280 -16537> 2013-04-03 10:37:51.657755 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29021 update_from_paxos applying incremental 29022 -16536> 2013-04-03 10:37:51.657770 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 v29022: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1615B/s wr, 0op/s -16535> 2013-04-03 10:37:51.657787 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 send_pg_creates to 0 pgs -16534> 2013-04-03 10:37:51.657790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 update_logger -16533> 2013-04-03 10:37:51.657799 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 create_pending v 29023 -16532> 2013-04-03 10:37:51.657808 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29022 _updated_stats for osd.1 10.252.24.25:6805/3572 -16531> 2013-04-03 10:37:51.657821 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 22) v1 tid 17521) v2 -- ?+0 0x24f8b40 con 0x208a840 -16530> 2013-04-03 10:37:51.657831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 check_osd_map already seen 605 >= 605 -16529> 2013-04-03 10:37:51.657833 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 update_logger -16528> 2013-04-03 10:37:51.657844 7fcdffcdb700 0 log [INF] : pgmap v29022: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1615B/s wr, 0op/s -16527> 2013-04-03 10:37:51.657845 7fcdffcdb700 10 send_log to self -16526> 2013-04-03 10:37:51.657846 7fcdffcdb700 10 log_queue is 1 last_log 88 sent 87 num 1 unsent 1 sending 1 -16525> 2013-04-03 10:37:51.657853 7fcdffcdb700 10 will send 2013-04-03 10:37:51.657845 mon.0 10.252.24.25:6789/0 88 : [INF] pgmap v29022: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1615B/s wr, 0op/s -16524> 2013-04-03 10:37:51.657860 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287ec00 -16523> 2013-04-03 10:37:51.657906 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos -16522> 2013-04-03 10:37:51.657925 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 -16521> 2013-04-03 10:37:51.657941 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16520> 2013-04-03 10:37:51.657962 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287ec00 con 0x1bac160 -16519> 2013-04-03 10:37:51.657983 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos -16518> 2013-04-03 10:37:51.657998 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 -16517> 2013-04-03 10:37:51.658005 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16516> 2013-04-03 10:37:51.658015 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 preprocess_log log(1 entries) v1 from mon.0 -16515> 2013-04-03 10:37:51.658037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16514> 2013-04-03 10:37:51.658049 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 prepare_log log(1 entries) v1 from mon.0 -16513> 2013-04-03 10:37:51.658058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 logging 2013-04-03 10:37:51.657845 mon.0 10.252.24.25:6789/0 88 : [INF] pgmap v29022: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1615B/s wr, 0op/s -16512> 2013-04-03 10:37:51.669009 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113329 ==== paxos(lease_ack lc 427330 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (2725069382 0 0) 0x26dc280 con 0x208a840 -16511> 2013-04-03 10:37:51.669044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos -16510> 2013-04-03 10:37:51.669059 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394025 update_from_paxos version 394025 summary v 394025 -16509> 2013-04-03 10:37:51.669074 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16508> 2013-04-03 10:37:51.743456 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113330 ==== forward(pg_stats(1 pgs tid 30442 v 605) v1) to leader v1 ==== 757+0+0 (3201538102 0 0) 0x295e340 con 0x208a840 -16507> 2013-04-03 10:37:51.743465 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -16506> 2013-04-03 10:37:51.743474 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -16505> 2013-04-03 10:37:51.743499 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 preprocess_query pg_stats(1 pgs tid 30442 v 605) v1 from osd.3 10.252.24.88:6801/17538 -16504> 2013-04-03 10:37:51.743504 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 prepare_update pg_stats(1 pgs tid 30442 v 605) v1 from osd.3 10.252.24.88:6801/17538 -16503> 2013-04-03 10:37:51.743507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 prepare_pg_stats pg_stats(1 pgs tid 30442 v 605) v1 from osd.3 -16502> 2013-04-03 10:37:51.743511 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29022 got osd.3 osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -16501> 2013-04-03 10:37:52.657222 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29022 encode_pending v 29023 -16500> 2013-04-03 10:37:52.657254 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427289..427330) queue_proposal bl 654 bytes; ctx = 0x1b91fc0 -16499> 2013-04-03 10:37:52.657261 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427289..427330) propose_queued 427331 654 bytes -16498> 2013-04-03 10:37:52.658412 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427330 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0a00 -16497> 2013-04-03 10:37:52.658518 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394025 encode_pending v394026 -16496> 2013-04-03 10:37:52.658544 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427289..427330) queue_proposal bl 443 bytes; ctx = 0x1b91e00 -16495> 2013-04-03 10:37:52.658549 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427289..427330) propose_new_value not active; proposal queued -16494> 2013-04-03 10:37:52.684851 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113331 ==== paxos(accept lc 427330 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2672755255 0 0) 0x29e0a00 con 0x208a840 -16493> 2013-04-03 10:37:52.685203 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427331 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc280 -16492> 2013-04-03 10:37:52.685228 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427331 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29e0780 -16491> 2013-04-03 10:37:52.685299 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29022 update_from_paxos applying incremental 29023 -16490> 2013-04-03 10:37:52.685319 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 v29023: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -16489> 2013-04-03 10:37:52.685338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 send_pg_creates to 0 pgs -16488> 2013-04-03 10:37:52.685341 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 update_logger -16487> 2013-04-03 10:37:52.685350 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 create_pending v 29024 -16486> 2013-04-03 10:37:52.685359 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29023 _updated_stats for osd.3 10.252.24.88:6801/17538 -16485> 2013-04-03 10:37:52.685364 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30442) v1 tid 17522) v2 -- ?+0 0x24f8900 con 0x208a840 -16484> 2013-04-03 10:37:52.685376 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 check_osd_map already seen 605 >= 605 -16483> 2013-04-03 10:37:52.685378 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 update_logger -16482> 2013-04-03 10:37:52.685388 7fcdffcdb700 0 log [INF] : pgmap v29023: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -16481> 2013-04-03 10:37:52.685389 7fcdffcdb700 10 send_log to self -16480> 2013-04-03 10:37:52.685395 7fcdffcdb700 10 log_queue is 2 last_log 89 sent 88 num 2 unsent 1 sending 1 -16479> 2013-04-03 10:37:52.685397 7fcdffcdb700 10 will send 2013-04-03 10:37:51.657845 mon.0 10.252.24.25:6789/0 88 : [INF] pgmap v29022: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1615B/s wr, 0op/s -16478> 2013-04-03 10:37:52.685403 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b5b000 -16477> 2013-04-03 10:37:52.685429 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427289..427331) propose_queued 427332 443 bytes -16476> 2013-04-03 10:37:52.685652 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427331 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4280 -16475> 2013-04-03 10:37:52.685686 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b5b000 con 0x1bac160 -16474> 2013-04-03 10:37:52.696825 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113332 ==== paxos(lease_ack lc 427331 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (26230013 0 0) 0x29b4280 con 0x208a840 -16473> 2013-04-03 10:37:52.710024 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113333 ==== paxos(accept lc 427331 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3789250229 0 0) 0x29e0780 con 0x208a840 -16472> 2013-04-03 10:37:52.710306 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427332 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4280 -16471> 2013-04-03 10:37:52.710322 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427332 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29e0a00 -16470> 2013-04-03 10:37:52.710354 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos -16469> 2013-04-03 10:37:52.710378 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394025 -16468> 2013-04-03 10:37:52.710392 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos latest full 393798 -16467> 2013-04-03 10:37:52.710409 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394026 update_from_paxos applying incremental log 394026 2013-04-03 10:37:51.657845 mon.0 10.252.24.25:6789/0 88 : [INF] pgmap v29022: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1615B/s wr, 0op/s -16466> 2013-04-03 10:37:52.710453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 check_subs -16465> 2013-04-03 10:37:52.710483 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 create_pending v 394027 -16464> 2013-04-03 10:37:52.710505 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394026 _updated_log for mon.0 10.252.24.25:6789/0 -16463> 2013-04-03 10:37:52.710517 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 88) v1 -- ?+0 0x2224380 con 0x1bac160 -16462> 2013-04-03 10:37:52.710537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos -16461> 2013-04-03 10:37:52.710553 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 -16460> 2013-04-03 10:37:52.710562 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16459> 2013-04-03 10:37:52.710573 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 preprocess_log log(1 entries) v1 from mon.0 -16458> 2013-04-03 10:37:52.710589 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 nothing new -16457> 2013-04-03 10:37:52.710636 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos -16456> 2013-04-03 10:37:52.710651 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 -16455> 2013-04-03 10:37:52.710669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16454> 2013-04-03 10:37:52.710689 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 88) v1 ==== 0+0+0 (0 0 0) 0x2224380 con 0x1bac160 -16453> 2013-04-03 10:37:52.710697 7fcdffcdb700 10 handle_log_ack log(last 88) v1 -16452> 2013-04-03 10:37:52.710699 7fcdffcdb700 10 logged 2013-04-03 10:37:51.657845 mon.0 10.252.24.25:6789/0 88 : [INF] pgmap v29022: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1615B/s wr, 0op/s -16451> 2013-04-03 10:37:52.721934 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113334 ==== paxos(lease_ack lc 427332 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (1180112263 0 0) 0x29e0a00 con 0x208a840 -16450> 2013-04-03 10:37:52.721969 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos -16449> 2013-04-03 10:37:52.721985 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 -16448> 2013-04-03 10:37:52.721999 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16447> 2013-04-03 10:37:52.928983 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113335 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 66 v187) v2) to leader v1 ==== 560+0+0 (3185799799 0 0) 0x2a51b00 con 0x208a840 -16446> 2013-04-03 10:37:52.928998 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -16445> 2013-04-03 10:37:52.929006 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52600 from 10.252.24.88:6789/0 -16444> 2013-04-03 10:37:52.929036 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 66 v187) v2 from mds.0 10.252.24.25:6802/3144 -16443> 2013-04-03 10:37:52.929047 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 66 v187) v2 tid 17523) v2 -- ?+0 0x24f8b40 con 0x208a840 -16442> 2013-04-03 10:37:53.337529 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29023 check_down_pgs -16441> 2013-04-03 10:37:53.337563 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29023 v29023: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -16440> 2013-04-03 10:37:53.337872 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -16439> 2013-04-03 10:37:53.337896 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -16438> 2013-04-03 10:37:53.337933 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -16437> 2013-04-03 10:37:53.337964 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos -16436> 2013-04-03 10:37:53.337983 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 -16435> 2013-04-03 10:37:53.337990 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394026 log -16434> 2013-04-03 10:37:53.338006 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16433> 2013-04-03 10:37:53.338024 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -16432> 2013-04-03 10:37:53.382220 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 64 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21722 v187) v2 ==== 289+0+0 (126832974 0 0) 0x2996b80 con 0x200d420 -16431> 2013-04-03 10:37:53.382272 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21722 v187) v2 from mds.? 10.252.24.88:6800/17213 -16430> 2013-04-03 10:37:53.382283 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21722 v187) v2 -- ?+0 0x2a51b00 con 0x200d420 -16429> 2013-04-03 10:37:55.582789 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 65 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140352 v187) v2 ==== 289+0+0 (2689677490 0 0) 0x2a83840 con 0x2118420 -16428> 2013-04-03 10:37:55.582862 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140352 v187) v2 from mds.? 10.252.58.31:6800/10908 -16427> 2013-04-03 10:37:55.582875 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140352 v187) v2 -- ?+0 0x2996b80 con 0x2118420 -16426> 2013-04-03 10:37:55.710396 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427332 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29e0c80 -16425> 2013-04-03 10:37:55.711151 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113336 ==== paxos(lease_ack lc 427332 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (2571027384 0 0) 0x29e0c80 con 0x208a840 -16424> 2013-04-03 10:37:55.711202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos -16423> 2013-04-03 10:37:55.711221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 -16422> 2013-04-03 10:37:55.711238 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16421> 2013-04-03 10:37:56.928935 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113337 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 67 v187) v2) to leader v1 ==== 560+0+0 (423586857 0 0) 0x295e340 con 0x208a840 -16420> 2013-04-03 10:37:56.928958 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -16419> 2013-04-03 10:37:56.928968 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52b80 from 10.252.24.88:6789/0 -16418> 2013-04-03 10:37:56.929019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 67 v187) v2 from mds.0 10.252.24.25:6802/3144 -16417> 2013-04-03 10:37:56.929031 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 67 v187) v2 tid 17524) v2 -- ?+0 0x24f8900 con 0x208a840 -16416> 2013-04-03 10:37:57.382368 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 65 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21723 v187) v2 ==== 289+0+0 (618786355 0 0) 0x2996340 con 0x200d420 -16415> 2013-04-03 10:37:57.382445 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21723 v187) v2 from mds.? 10.252.24.88:6800/17213 -16414> 2013-04-03 10:37:57.382458 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21723 v187) v2 -- ?+0 0x295e340 con 0x200d420 -16413> 2013-04-03 10:37:58.215632 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113338 ==== forward(pg_stats(1 pgs tid 4 v 605) v1) to leader v1 ==== 757+0+0 (508889349 0 0) 0x2a83840 con 0x208a840 -16412> 2013-04-03 10:37:58.215661 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -16411> 2013-04-03 10:37:58.215671 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -16410> 2013-04-03 10:37:58.215723 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 preprocess_query pg_stats(1 pgs tid 4 v 605) v1 from osd.2 10.252.24.30:6801/5656 -16409> 2013-04-03 10:37:58.215729 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 prepare_update pg_stats(1 pgs tid 4 v 605) v1 from osd.2 10.252.24.30:6801/5656 -16408> 2013-04-03 10:37:58.215732 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 prepare_pg_stats pg_stats(1 pgs tid 4 v 605) v1 from osd.2 -16407> 2013-04-03 10:37:58.215739 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29023 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -16406> 2013-04-03 10:37:58.265925 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29023 encode_pending v 29024 -16405> 2013-04-03 10:37:58.265959 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427289..427332) queue_proposal bl 654 bytes; ctx = 0x1b91fd0 -16404> 2013-04-03 10:37:58.265968 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427289..427332) propose_queued 427333 654 bytes -16403> 2013-04-03 10:37:58.267015 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427332 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1780 -16402> 2013-04-03 10:37:58.292143 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113339 ==== paxos(accept lc 427332 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1029115937 0 0) 0x29a1780 con 0x208a840 -16401> 2013-04-03 10:37:58.293007 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427333 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0c80 -16400> 2013-04-03 10:37:58.293039 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427333 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29e0a00 -16399> 2013-04-03 10:37:58.293112 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29023 update_from_paxos applying incremental 29024 -16398> 2013-04-03 10:37:58.293128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 v29024: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3239B/s wr, 0op/s -16397> 2013-04-03 10:37:58.293145 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 send_pg_creates to 0 pgs -16396> 2013-04-03 10:37:58.293148 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 update_logger -16395> 2013-04-03 10:37:58.293157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 create_pending v 29025 -16394> 2013-04-03 10:37:58.293166 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29024 _updated_stats for osd.2 10.252.24.30:6801/5656 -16393> 2013-04-03 10:37:58.293171 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 4) v1 tid 17525) v2 -- ?+0 0x24f8b40 con 0x208a840 -16392> 2013-04-03 10:37:58.293184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 check_osd_map already seen 605 >= 605 -16391> 2013-04-03 10:37:58.293186 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 update_logger -16390> 2013-04-03 10:37:58.293196 7fcdffcdb700 0 log [INF] : pgmap v29024: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3239B/s wr, 0op/s -16389> 2013-04-03 10:37:58.293203 7fcdffcdb700 10 send_log to self -16388> 2013-04-03 10:37:58.293203 7fcdffcdb700 10 log_queue is 2 last_log 90 sent 88 num 2 unsent 2 sending 2 -16387> 2013-04-03 10:37:58.293205 7fcdffcdb700 10 will send 2013-04-03 10:37:52.685389 mon.0 10.252.24.25:6789/0 89 : [INF] pgmap v29023: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -16386> 2013-04-03 10:37:58.293211 7fcdffcdb700 10 will send 2013-04-03 10:37:58.293202 mon.0 10.252.24.25:6789/0 90 : [INF] pgmap v29024: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3239B/s wr, 0op/s -16385> 2013-04-03 10:37:58.293218 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b5b000 -16384> 2013-04-03 10:37:58.293269 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos -16383> 2013-04-03 10:37:58.293288 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 -16382> 2013-04-03 10:37:58.293305 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16381> 2013-04-03 10:37:58.293334 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b5b000 con 0x1bac160 -16380> 2013-04-03 10:37:58.293357 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos -16379> 2013-04-03 10:37:58.293371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 -16378> 2013-04-03 10:37:58.293379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -16377> 2013-04-03 10:37:58.293389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 preprocess_log log(2 entries) v1 from mon.0 -16376> 2013-04-03 10:37:58.293418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -16375> 2013-04-03 10:37:58.293429 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 prepare_log log(2 entries) v1 from mon.0 -16374> 2013-04-03 10:37:58.293439 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 logging 2013-04-03 10:37:52.685389 mon.0 10.252.24.25:6789/0 89 : [INF] pgmap v29023: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -16373> 2013-04-03 10:37:58.293458 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 logging 2013-04-03 10:37:58.293202 mon.0 10.252.24.25:6789/0 90 : [INF] pgmap v29024: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3239B/s wr, 0op/s -16372> 2013-04-03 10:37:58.317434 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113340 ==== paxos(lease_ack lc 427333 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (867291749 0 0) 0x29e0a00 con 0x208a840 -16371> 2013-04-03 10:37:58.317482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos -16370> 2013-04-03 10:37:58.317502 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 -16369> 2013-04-03 10:37:58.317517 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16368> 2013-04-03 10:37:58.338135 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29024 check_down_pgs -16367> 2013-04-03 10:37:58.338169 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29024 v29024: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3239B/s wr, 0op/s -16366> 2013-04-03 10:37:58.338194 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -16365> 2013-04-03 10:37:58.338215 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -16364> 2013-04-03 10:37:58.338251 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -16363> 2013-04-03 10:37:58.338281 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos -16362> 2013-04-03 10:37:58.338297 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394026 update_from_paxos version 394026 summary v 394026 -16361> 2013-04-03 10:37:58.338306 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394026 log -16360> 2013-04-03 10:37:58.338451 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16359> 2013-04-03 10:37:58.338474 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -16358> 2013-04-03 10:37:59.292347 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394026 encode_pending v394027 -16357> 2013-04-03 10:37:59.292375 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427289..427333) queue_proposal bl 725 bytes; ctx = 0x1b91fb0 -16356> 2013-04-03 10:37:59.292383 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427289..427333) propose_queued 427334 725 bytes -16355> 2013-04-03 10:37:59.293484 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427333 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1a00 -16354> 2013-04-03 10:37:59.341582 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113341 ==== paxos(accept lc 427333 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (913879688 0 0) 0x29a1a00 con 0x208a840 -16353> 2013-04-03 10:37:59.341887 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427334 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0a00 -16352> 2013-04-03 10:37:59.341925 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427334 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29a1780 -16351> 2013-04-03 10:37:59.341961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos -16350> 2013-04-03 10:37:59.341990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394026 -16349> 2013-04-03 10:37:59.342004 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos latest full 393798 -16348> 2013-04-03 10:37:59.342020 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394027 update_from_paxos applying incremental log 394027 2013-04-03 10:37:52.685389 mon.0 10.252.24.25:6789/0 89 : [INF] pgmap v29023: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -16347> 2013-04-03 10:37:59.342048 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394027 update_from_paxos applying incremental log 394027 2013-04-03 10:37:58.293202 mon.0 10.252.24.25:6789/0 90 : [INF] pgmap v29024: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3239B/s wr, 0op/s -16346> 2013-04-03 10:37:59.342086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 check_subs -16345> 2013-04-03 10:37:59.342117 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 create_pending v 394028 -16344> 2013-04-03 10:37:59.342139 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394027 _updated_log for mon.0 10.252.24.25:6789/0 -16343> 2013-04-03 10:37:59.342151 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 90) v1 -- ?+0 0x22948c0 con 0x1bac160 -16342> 2013-04-03 10:37:59.342202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos -16341> 2013-04-03 10:37:59.342218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 -16340> 2013-04-03 10:37:59.342237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16339> 2013-04-03 10:37:59.342260 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 90) v1 ==== 0+0+0 (0 0 0) 0x22948c0 con 0x1bac160 -16338> 2013-04-03 10:37:59.342268 7fcdffcdb700 10 handle_log_ack log(last 90) v1 -16337> 2013-04-03 10:37:59.342271 7fcdffcdb700 10 logged 2013-04-03 10:37:52.685389 mon.0 10.252.24.25:6789/0 89 : [INF] pgmap v29023: 240 pgs: 240 active+clean; 253 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -16336> 2013-04-03 10:37:59.342278 7fcdffcdb700 10 logged 2013-04-03 10:37:58.293202 mon.0 10.252.24.25:6789/0 90 : [INF] pgmap v29024: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3239B/s wr, 0op/s -16335> 2013-04-03 10:37:59.353494 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113342 ==== paxos(lease_ack lc 427334 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (3786305629 0 0) 0x29a1780 con 0x208a840 -16334> 2013-04-03 10:37:59.353531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos -16333> 2013-04-03 10:37:59.353547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 -16332> 2013-04-03 10:37:59.353561 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16331> 2013-04-03 10:37:59.582897 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 66 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140353 v187) v2 ==== 289+0+0 (2201978831 0 0) 0x2a83dc0 con 0x2118420 -16330> 2013-04-03 10:37:59.582944 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140353 v187) v2 from mds.? 10.252.58.31:6800/10908 -16329> 2013-04-03 10:37:59.582955 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140353 v187) v2 -- ?+0 0x2a84080 con 0x2118420 -16328> 2013-04-03 10:38:00.441184 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2acda00 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33057/0 -16327> 2013-04-03 10:38:00.441216 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acda00 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33057/0) -16326> 2013-04-03 10:38:00.441233 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acda00 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -16325> 2013-04-03 10:38:00.441419 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acda00 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -16324> 2013-04-03 10:38:00.929012 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113343 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 68 v187) v2) to leader v1 ==== 560+0+0 (1584098603 0 0) 0x2a83b00 con 0x208a840 -16323> 2013-04-03 10:38:00.929030 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -16322> 2013-04-03 10:38:00.929038 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2996340 from 10.252.24.88:6789/0 -16321> 2013-04-03 10:38:00.929089 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 68 v187) v2 from mds.0 10.252.24.25:6802/3144 -16320> 2013-04-03 10:38:00.929101 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 68 v187) v2 tid 17526) v2 -- ?+0 0x2b26b40 con 0x208a840 -16319> 2013-04-03 10:38:01.382495 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 66 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21724 v187) v2 ==== 289+0+0 (3437810240 0 0) 0x2a83840 con 0x200d420 -16318> 2013-04-03 10:38:01.382565 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21724 v187) v2 from mds.? 10.252.24.88:6800/17213 -16317> 2013-04-03 10:38:01.382578 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21724 v187) v2 -- ?+0 0x2a83b00 con 0x200d420 -16316> 2013-04-03 10:38:02.342001 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427334 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29fd780 -16315> 2013-04-03 10:38:02.342815 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113344 ==== paxos(lease_ack lc 427334 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (3264418643 0 0) 0x29fd780 con 0x208a840 -16314> 2013-04-03 10:38:02.342889 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos -16313> 2013-04-03 10:38:02.342911 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 -16312> 2013-04-03 10:38:02.342928 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16311> 2013-04-03 10:38:03.338613 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29024 v29024: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3239B/s wr, 0op/s -16310> 2013-04-03 10:38:03.338684 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -16309> 2013-04-03 10:38:03.338707 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -16308> 2013-04-03 10:38:03.338749 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -16307> 2013-04-03 10:38:03.338780 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos -16306> 2013-04-03 10:38:03.338800 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 -16305> 2013-04-03 10:38:03.338808 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394027 log -16304> 2013-04-03 10:38:03.338826 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16303> 2013-04-03 10:38:03.338844 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -16302> 2013-04-03 10:38:03.583030 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 67 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140354 v187) v2 ==== 289+0+0 (3867952712 0 0) 0x29c58c0 con 0x2118420 -16301> 2013-04-03 10:38:03.583079 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140354 v187) v2 from mds.? 10.252.58.31:6800/10908 -16300> 2013-04-03 10:38:03.583091 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140354 v187) v2 -- ?+0 0x2a83840 con 0x2118420 -16299> 2013-04-03 10:38:04.929181 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113345 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 69 v187) v2) to leader v1 ==== 560+0+0 (1461891880 0 0) 0x2a83dc0 con 0x208a840 -16298> 2013-04-03 10:38:04.929203 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -16297> 2013-04-03 10:38:04.929212 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2995000 from 10.252.24.88:6789/0 -16296> 2013-04-03 10:38:04.929264 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 69 v187) v2 from mds.0 10.252.24.25:6802/3144 -16295> 2013-04-03 10:38:04.929274 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 69 v187) v2 tid 17527) v2 -- ?+0 0x24f8b40 con 0x208a840 -16294> 2013-04-03 10:38:05.342073 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427334 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29fd500 -16293> 2013-04-03 10:38:05.342816 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113346 ==== paxos(lease_ack lc 427334 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (3375649492 0 0) 0x29fd500 con 0x208a840 -16292> 2013-04-03 10:38:05.342888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos -16291> 2013-04-03 10:38:05.342908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 -16290> 2013-04-03 10:38:05.342924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16289> 2013-04-03 10:38:05.382565 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 67 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21725 v187) v2 ==== 289+0+0 (4018564413 0 0) 0x2a832c0 con 0x200d420 -16288> 2013-04-03 10:38:05.382612 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21725 v187) v2 from mds.? 10.252.24.88:6800/17213 -16287> 2013-04-03 10:38:05.382623 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21725 v187) v2 -- ?+0 0x2a83dc0 con 0x200d420 -16286> 2013-04-03 10:38:06.579101 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113347 ==== forward(pg_stats(1 pgs tid 23 v 605) v1) to leader v1 ==== 753+0+0 (713361198 0 0) 0x29c58c0 con 0x208a840 -16285> 2013-04-03 10:38:06.579122 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -16284> 2013-04-03 10:38:06.579132 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -16283> 2013-04-03 10:38:06.579183 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 preprocess_query pg_stats(1 pgs tid 23 v 605) v1 from osd.1 10.252.24.25:6805/3572 -16282> 2013-04-03 10:38:06.579206 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 prepare_update pg_stats(1 pgs tid 23 v 605) v1 from osd.1 10.252.24.25:6805/3572 -16281> 2013-04-03 10:38:06.579212 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 prepare_pg_stats pg_stats(1 pgs tid 23 v 605) v1 from osd.1 -16280> 2013-04-03 10:38:06.579218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29024 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -16279> 2013-04-03 10:38:06.629407 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29024 encode_pending v 29025 -16278> 2013-04-03 10:38:06.629733 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427289..427334) queue_proposal bl 650 bytes; ctx = 0x1b91ea0 -16277> 2013-04-03 10:38:06.629746 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427289..427334) propose_queued 427335 650 bytes -16276> 2013-04-03 10:38:06.645584 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427334 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fda00 -16275> 2013-04-03 10:38:06.763136 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113348 ==== paxos(accept lc 427334 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (890162163 0 0) 0x29fda00 con 0x208a840 -16274> 2013-04-03 10:38:06.763506 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427335 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd500 -16273> 2013-04-03 10:38:06.763538 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427335 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29fd780 -16272> 2013-04-03 10:38:06.763618 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29024 update_from_paxos applying incremental 29025 -16271> 2013-04-03 10:38:06.763638 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 v29025: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s -16270> 2013-04-03 10:38:06.763656 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 send_pg_creates to 0 pgs -16269> 2013-04-03 10:38:06.763659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 update_logger -16268> 2013-04-03 10:38:06.763669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 create_pending v 29026 -16267> 2013-04-03 10:38:06.763678 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29025 _updated_stats for osd.1 10.252.24.25:6805/3572 -16266> 2013-04-03 10:38:06.763684 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 23) v1 tid 17528) v2 -- ?+0 0x2b26b40 con 0x208a840 -16265> 2013-04-03 10:38:06.763698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 check_osd_map already seen 605 >= 605 -16264> 2013-04-03 10:38:06.763700 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 update_logger -16263> 2013-04-03 10:38:06.763710 7fcdffcdb700 0 log [INF] : pgmap v29025: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s -16262> 2013-04-03 10:38:06.763712 7fcdffcdb700 10 send_log to self -16261> 2013-04-03 10:38:06.763712 7fcdffcdb700 10 log_queue is 1 last_log 91 sent 90 num 1 unsent 1 sending 1 -16260> 2013-04-03 10:38:06.763714 7fcdffcdb700 10 will send 2013-04-03 10:38:06.763711 mon.0 10.252.24.25:6789/0 91 : [INF] pgmap v29025: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s -16259> 2013-04-03 10:38:06.763721 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b5b000 -16258> 2013-04-03 10:38:06.763770 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos -16257> 2013-04-03 10:38:06.763789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 -16256> 2013-04-03 10:38:06.763805 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16255> 2013-04-03 10:38:06.763829 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b5b000 con 0x1bac160 -16254> 2013-04-03 10:38:06.763855 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos -16253> 2013-04-03 10:38:06.763869 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 -16252> 2013-04-03 10:38:06.763877 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16251> 2013-04-03 10:38:06.763887 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 preprocess_log log(1 entries) v1 from mon.0 -16250> 2013-04-03 10:38:06.763909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16249> 2013-04-03 10:38:06.763921 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 prepare_log log(1 entries) v1 from mon.0 -16248> 2013-04-03 10:38:06.763931 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 logging 2013-04-03 10:38:06.763711 mon.0 10.252.24.25:6789/0 91 : [INF] pgmap v29025: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s -16247> 2013-04-03 10:38:06.776284 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113349 ==== paxos(lease_ack lc 427335 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (218487301 0 0) 0x29fd780 con 0x208a840 -16246> 2013-04-03 10:38:06.776327 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos -16245> 2013-04-03 10:38:06.776344 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394027 update_from_paxos version 394027 summary v 394027 -16244> 2013-04-03 10:38:06.776358 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16243> 2013-04-03 10:38:07.583148 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 68 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140355 v187) v2 ==== 289+0+0 (3319986485 0 0) 0x273adc0 con 0x2118420 -16242> 2013-04-03 10:38:07.583224 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140355 v187) v2 from mds.? 10.252.58.31:6800/10908 -16241> 2013-04-03 10:38:07.583237 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140355 v187) v2 -- ?+0 0x29c58c0 con 0x2118420 -16240> 2013-04-03 10:38:07.763333 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394027 encode_pending v394028 -16239> 2013-04-03 10:38:07.763358 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427289..427335) queue_proposal bl 443 bytes; ctx = 0x1b91f90 -16238> 2013-04-03 10:38:07.763364 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427289..427335) propose_queued 427336 443 bytes -16237> 2013-04-03 10:38:07.765076 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427335 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd280 -16236> 2013-04-03 10:38:07.949301 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113350 ==== paxos(accept lc 427335 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1122098064 0 0) 0x29fd280 con 0x208a840 -16235> 2013-04-03 10:38:07.949638 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427336 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd780 -16234> 2013-04-03 10:38:07.949663 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427336 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29fda00 -16233> 2013-04-03 10:38:07.949705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos -16232> 2013-04-03 10:38:07.949730 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394027 -16231> 2013-04-03 10:38:07.949744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos latest full 393798 -16230> 2013-04-03 10:38:07.949761 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394028 update_from_paxos applying incremental log 394028 2013-04-03 10:38:06.763711 mon.0 10.252.24.25:6789/0 91 : [INF] pgmap v29025: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s -16229> 2013-04-03 10:38:07.949807 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 check_subs -16228> 2013-04-03 10:38:07.949846 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 create_pending v 394029 -16227> 2013-04-03 10:38:07.949869 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394028 _updated_log for mon.0 10.252.24.25:6789/0 -16226> 2013-04-03 10:38:07.949881 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 91) v1 -- ?+0 0x2294e00 con 0x1bac160 -16225> 2013-04-03 10:38:07.949929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos -16224> 2013-04-03 10:38:07.949945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 -16223> 2013-04-03 10:38:07.949966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16222> 2013-04-03 10:38:07.949987 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 91) v1 ==== 0+0+0 (0 0 0) 0x2294e00 con 0x1bac160 -16221> 2013-04-03 10:38:07.949995 7fcdffcdb700 10 handle_log_ack log(last 91) v1 -16220> 2013-04-03 10:38:07.949997 7fcdffcdb700 10 logged 2013-04-03 10:38:06.763711 mon.0 10.252.24.25:6789/0 91 : [INF] pgmap v29025: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s -16219> 2013-04-03 10:38:07.962509 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113351 ==== paxos(lease_ack lc 427336 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (529106687 0 0) 0x29fda00 con 0x208a840 -16218> 2013-04-03 10:38:07.962547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos -16217> 2013-04-03 10:38:07.962563 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 -16216> 2013-04-03 10:38:07.962577 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16215> 2013-04-03 10:38:08.338986 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29025 check_down_pgs -16214> 2013-04-03 10:38:08.339016 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29025 v29025: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s -16213> 2013-04-03 10:38:08.339043 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -16212> 2013-04-03 10:38:08.339064 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -16211> 2013-04-03 10:38:08.339101 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -16210> 2013-04-03 10:38:08.339132 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos -16209> 2013-04-03 10:38:08.339154 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 -16208> 2013-04-03 10:38:08.339163 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394028 log -16207> 2013-04-03 10:38:08.339180 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16206> 2013-04-03 10:38:08.339198 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -16205> 2013-04-03 10:38:08.929294 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113352 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 70 v187) v2) to leader v1 ==== 560+0+0 (1579904596 0 0) 0x2995dc0 con 0x208a840 -16204> 2013-04-03 10:38:08.929311 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -16203> 2013-04-03 10:38:08.929320 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2995580 from 10.252.24.88:6789/0 -16202> 2013-04-03 10:38:08.929358 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 70 v187) v2 from mds.0 10.252.24.25:6802/3144 -16201> 2013-04-03 10:38:08.929369 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 70 v187) v2 tid 17529) v2 -- ?+0 0x24f8b40 con 0x208a840 -16200> 2013-04-03 10:38:09.193435 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427289..427336) queue_proposal bl 1167 bytes; ctx = 0x1b91eb0 -16199> 2013-04-03 10:38:09.382692 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 68 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21726 v187) v2 ==== 289+0+0 (2318773434 0 0) 0x2a84b80 con 0x200d420 -16198> 2013-04-03 10:38:09.382768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21726 v187) v2 from mds.? 10.252.24.88:6800/17213 -16197> 2013-04-03 10:38:09.382781 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21726 v187) v2 -- ?+0 0x2995dc0 con 0x200d420 -16196> 2013-04-03 10:38:10.949737 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427336 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29fd000 -16195> 2013-04-03 10:38:10.950486 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113353 ==== paxos(lease_ack lc 427336 fc 427289 pn 0 opn 0) v3 ==== 80+0+0 (1319224644 0 0) 0x29fd000 con 0x208a840 -16194> 2013-04-03 10:38:10.950552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos -16193> 2013-04-03 10:38:10.950696 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 -16192> 2013-04-03 10:38:10.950723 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16191> 2013-04-03 10:38:11.583294 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 69 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140356 v187) v2 ==== 289+0+0 (770426182 0 0) 0x273b8c0 con 0x2118420 -16190> 2013-04-03 10:38:11.583367 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140356 v187) v2 from mds.? 10.252.58.31:6800/10908 -16189> 2013-04-03 10:38:11.583379 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140356 v187) v2 -- ?+0 0x2a84b80 con 0x2118420 -16188> 2013-04-03 10:38:12.929447 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113354 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 71 v187) v2) to leader v1 ==== 560+0+0 (3477742131 0 0) 0x273adc0 con 0x208a840 -16187> 2013-04-03 10:38:12.929469 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -16186> 2013-04-03 10:38:12.929478 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29952c0 from 10.252.24.88:6789/0 -16185> 2013-04-03 10:38:12.929531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 71 v187) v2 from mds.0 10.252.24.25:6802/3144 -16184> 2013-04-03 10:38:12.929543 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 71 v187) v2 tid 17530) v2 -- ?+0 0x2b26b40 con 0x208a840 -16183> 2013-04-03 10:38:13.217785 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113355 ==== forward(pg_stats(1 pgs tid 5 v 605) v1) to leader v1 ==== 757+0+0 (3553565299 0 0) 0x273b8c0 con 0x208a840 -16182> 2013-04-03 10:38:13.217811 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -16181> 2013-04-03 10:38:13.217821 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 -16180> 2013-04-03 10:38:13.217872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 preprocess_query pg_stats(1 pgs tid 5 v 605) v1 from osd.2 10.252.24.30:6801/5656 -16179> 2013-04-03 10:38:13.217893 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 prepare_update pg_stats(1 pgs tid 5 v 605) v1 from osd.2 10.252.24.30:6801/5656 -16178> 2013-04-03 10:38:13.217897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 prepare_pg_stats pg_stats(1 pgs tid 5 v 605) v1 from osd.2 -16177> 2013-04-03 10:38:13.217904 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29025 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -16176> 2013-04-03 10:38:13.268105 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29025 encode_pending v 29026 -16175> 2013-04-03 10:38:13.268147 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427289..427336) queue_proposal bl 654 bytes; ctx = 0x1b91ff0 -16174> 2013-04-03 10:38:13.268157 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427289..427336) propose_queued 427337 1167 bytes -16173> 2013-04-03 10:38:13.269104 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427336 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2acda00 -16172> 2013-04-03 10:38:13.298887 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113356 ==== paxos(accept lc 427336 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3230855180 0 0) 0x2acda00 con 0x208a840 -16171> 2013-04-03 10:38:13.299248 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427337 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd000 -16170> 2013-04-03 10:38:13.299272 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427337 fc 427289 pn 0 opn 0) v3 -- ?+0 0x29fda00 -16169> 2013-04-03 10:38:13.299333 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427337) propose_queued 427338 654 bytes -16168> 2013-04-03 10:38:13.299533 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427337 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd280 -16167> 2013-04-03 10:38:13.310794 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113357 ==== paxos(lease_ack lc 427337 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1034373351 0 0) 0x29fd280 con 0x208a840 -16166> 2013-04-03 10:38:13.323949 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113358 ==== paxos(accept lc 427337 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2807929007 0 0) 0x29fda00 con 0x208a840 -16165> 2013-04-03 10:38:13.324256 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427338 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd280 -16164> 2013-04-03 10:38:13.324288 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427338 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2acda00 -16163> 2013-04-03 10:38:13.324350 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29025 update_from_paxos applying incremental 29026 -16162> 2013-04-03 10:38:13.324369 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 v29026: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -16161> 2013-04-03 10:38:13.324387 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 send_pg_creates to 0 pgs -16160> 2013-04-03 10:38:13.324391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 update_logger -16159> 2013-04-03 10:38:13.324401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 create_pending v 29027 -16158> 2013-04-03 10:38:13.324410 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29026 _updated_stats for osd.2 10.252.24.30:6801/5656 -16157> 2013-04-03 10:38:13.324415 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 5) v1 tid 17531) v2 -- ?+0 0x24f8b40 con 0x208a840 -16156> 2013-04-03 10:38:13.324425 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 check_osd_map already seen 605 >= 605 -16155> 2013-04-03 10:38:13.324427 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 update_logger -16154> 2013-04-03 10:38:13.324437 7fcdffcdb700 0 log [INF] : pgmap v29026: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -16153> 2013-04-03 10:38:13.324439 7fcdffcdb700 10 send_log to self -16152> 2013-04-03 10:38:13.324439 7fcdffcdb700 10 log_queue is 1 last_log 92 sent 91 num 1 unsent 1 sending 1 -16151> 2013-04-03 10:38:13.324441 7fcdffcdb700 10 will send 2013-04-03 10:38:13.324438 mon.0 10.252.24.25:6789/0 92 : [INF] pgmap v29026: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -16150> 2013-04-03 10:38:13.324454 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b5b000 -16149> 2013-04-03 10:38:13.324502 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos -16148> 2013-04-03 10:38:13.324521 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 -16147> 2013-04-03 10:38:13.324539 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16146> 2013-04-03 10:38:13.324559 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b5b000 con 0x1bac160 -16145> 2013-04-03 10:38:13.324581 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos -16144> 2013-04-03 10:38:13.324596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 -16143> 2013-04-03 10:38:13.324605 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16142> 2013-04-03 10:38:13.324615 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 preprocess_log log(1 entries) v1 from mon.0 -16141> 2013-04-03 10:38:13.324638 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16140> 2013-04-03 10:38:13.324649 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 prepare_log log(1 entries) v1 from mon.0 -16139> 2013-04-03 10:38:13.324659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 logging 2013-04-03 10:38:13.324438 mon.0 10.252.24.25:6789/0 92 : [INF] pgmap v29026: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -16138> 2013-04-03 10:38:13.339313 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29026 check_down_pgs -16137> 2013-04-03 10:38:13.339348 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29026 v29026: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -16136> 2013-04-03 10:38:13.339369 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -16135> 2013-04-03 10:38:13.339390 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -16134> 2013-04-03 10:38:13.339426 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -16133> 2013-04-03 10:38:13.339456 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos -16132> 2013-04-03 10:38:13.339473 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 -16131> 2013-04-03 10:38:13.339482 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394028 log -16130> 2013-04-03 10:38:13.339497 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16129> 2013-04-03 10:38:13.339516 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -16128> 2013-04-03 10:38:13.349307 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113359 ==== paxos(lease_ack lc 427338 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3058789168 0 0) 0x2acda00 con 0x208a840 -16127> 2013-04-03 10:38:13.349351 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos -16126> 2013-04-03 10:38:13.349367 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394028 update_from_paxos version 394028 summary v 394028 -16125> 2013-04-03 10:38:13.349381 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16124> 2013-04-03 10:38:13.382768 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 69 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21727 v187) v2 ==== 289+0+0 (2841318343 0 0) 0x2a84340 con 0x200d420 -16123> 2013-04-03 10:38:13.382813 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21727 v187) v2 from mds.? 10.252.24.88:6800/17213 -16122> 2013-04-03 10:38:13.382823 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21727 v187) v2 -- ?+0 0x273b8c0 con 0x200d420 -16121> 2013-04-03 10:38:14.324147 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394028 encode_pending v394029 -16120> 2013-04-03 10:38:14.324176 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427338) queue_proposal bl 443 bytes; ctx = 0x1b91f80 -16119> 2013-04-03 10:38:14.324183 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427338) propose_queued 427339 443 bytes -16118> 2013-04-03 10:38:14.325406 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427338 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4dc80 -16117> 2013-04-03 10:38:14.440059 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113360 ==== paxos(accept lc 427338 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (768006887 0 0) 0x2c4dc80 con 0x208a840 -16116> 2013-04-03 10:38:14.440363 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427339 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2acda00 -16115> 2013-04-03 10:38:14.440399 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427339 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29fda00 -16114> 2013-04-03 10:38:14.440444 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos -16113> 2013-04-03 10:38:14.440470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394028 -16112> 2013-04-03 10:38:14.440484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos latest full 393798 -16111> 2013-04-03 10:38:14.440500 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394029 update_from_paxos applying incremental log 394029 2013-04-03 10:38:13.324438 mon.0 10.252.24.25:6789/0 92 : [INF] pgmap v29026: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -16110> 2013-04-03 10:38:14.440547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 check_subs -16109> 2013-04-03 10:38:14.440578 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 create_pending v 394030 -16108> 2013-04-03 10:38:14.440600 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394029 _updated_log for mon.0 10.252.24.25:6789/0 -16107> 2013-04-03 10:38:14.440613 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 92) v1 -- ?+0 0x2294540 con 0x1bac160 -16106> 2013-04-03 10:38:14.440663 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos -16105> 2013-04-03 10:38:14.440678 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 -16104> 2013-04-03 10:38:14.440696 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16103> 2013-04-03 10:38:14.440718 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 92) v1 ==== 0+0+0 (0 0 0) 0x2294540 con 0x1bac160 -16102> 2013-04-03 10:38:14.440726 7fcdffcdb700 10 handle_log_ack log(last 92) v1 -16101> 2013-04-03 10:38:14.440729 7fcdffcdb700 10 logged 2013-04-03 10:38:13.324438 mon.0 10.252.24.25:6789/0 92 : [INF] pgmap v29026: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -16100> 2013-04-03 10:38:14.451935 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113361 ==== paxos(lease_ack lc 427339 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (308932423 0 0) 0x29fda00 con 0x208a840 -16099> 2013-04-03 10:38:14.451970 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos -16098> 2013-04-03 10:38:14.451990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 -16097> 2013-04-03 10:38:14.452006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16096> 2013-04-03 10:38:15.583412 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 70 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140357 v187) v2 ==== 289+0+0 (243628603 0 0) 0x2a84b80 con 0x2118420 -16095> 2013-04-03 10:38:15.583482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140357 v187) v2 from mds.? 10.252.58.31:6800/10908 -16094> 2013-04-03 10:38:15.583494 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140357 v187) v2 -- ?+0 0x2a84340 con 0x2118420 -16093> 2013-04-03 10:38:16.929696 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113362 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 72 v187) v2) to leader v1 ==== 560+0+0 (2627151202 0 0) 0x2995840 con 0x208a840 -16092> 2013-04-03 10:38:16.929718 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -16091> 2013-04-03 10:38:16.929728 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52b80 from 10.252.24.88:6789/0 -16090> 2013-04-03 10:38:16.929778 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 72 v187) v2 from mds.0 10.252.24.25:6802/3144 -16089> 2013-04-03 10:38:16.929790 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 72 v187) v2 tid 17532) v2 -- ?+0 0x2b26b40 con 0x208a840 -16088> 2013-04-03 10:38:17.382950 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 70 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21728 v187) v2 ==== 289+0+0 (413043963 0 0) 0x2a84600 con 0x200d420 -16087> 2013-04-03 10:38:17.383026 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21728 v187) v2 from mds.? 10.252.24.88:6800/17213 -16086> 2013-04-03 10:38:17.383038 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21728 v187) v2 -- ?+0 0x2995840 con 0x200d420 -16085> 2013-04-03 10:38:17.440471 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427339 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c4da00 -16084> 2013-04-03 10:38:17.441373 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113363 ==== paxos(lease_ack lc 427339 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1789133838 0 0) 0x2c4da00 con 0x208a840 -16083> 2013-04-03 10:38:17.441424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos -16082> 2013-04-03 10:38:17.441443 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 -16081> 2013-04-03 10:38:17.441460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16080> 2013-04-03 10:38:18.339664 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29026 v29026: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -16079> 2013-04-03 10:38:18.339704 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -16078> 2013-04-03 10:38:18.339727 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -16077> 2013-04-03 10:38:18.339767 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -16076> 2013-04-03 10:38:18.339798 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos -16075> 2013-04-03 10:38:18.339816 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 -16074> 2013-04-03 10:38:18.339826 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394029 log -16073> 2013-04-03 10:38:18.339842 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16072> 2013-04-03 10:38:18.339861 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -16071> 2013-04-03 10:38:19.583530 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 71 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140358 v187) v2 ==== 289+0+0 (1798749116 0 0) 0x2a83840 con 0x2118420 -16070> 2013-04-03 10:38:19.583601 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140358 v187) v2 from mds.? 10.252.58.31:6800/10908 -16069> 2013-04-03 10:38:19.583619 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140358 v187) v2 -- ?+0 0x2a84600 con 0x2118420 -16068> 2013-04-03 10:38:20.440545 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427339 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c4d780 -16067> 2013-04-03 10:38:20.441315 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113364 ==== paxos(lease_ack lc 427339 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1471144405 0 0) 0x2c4d780 con 0x208a840 -16066> 2013-04-03 10:38:20.441387 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos -16065> 2013-04-03 10:38:20.441407 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 -16064> 2013-04-03 10:38:20.441424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16063> 2013-04-03 10:38:20.929635 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113365 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 73 v187) v2) to leader v1 ==== 560+0+0 (2514418529 0 0) 0x2a84b80 con 0x208a840 -16062> 2013-04-03 10:38:20.929652 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -16061> 2013-04-03 10:38:20.929660 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52600 from 10.252.24.88:6789/0 -16060> 2013-04-03 10:38:20.929693 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 73 v187) v2 from mds.0 10.252.24.25:6802/3144 -16059> 2013-04-03 10:38:20.929704 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 73 v187) v2 tid 17533) v2 -- ?+0 0x24f8b40 con 0x208a840 -16058> 2013-04-03 10:38:21.383141 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 71 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21729 v187) v2 ==== 289+0+0 (1005597574 0 0) 0x2a83000 con 0x200d420 -16057> 2013-04-03 10:38:21.383212 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21729 v187) v2 from mds.? 10.252.24.88:6800/17213 -16056> 2013-04-03 10:38:21.383224 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21729 v187) v2 -- ?+0 0x2a84b80 con 0x200d420 -16055> 2013-04-03 10:38:21.580812 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113366 ==== forward(pg_stats(1 pgs tid 24 v 605) v1) to leader v1 ==== 753+0+0 (2502802800 0 0) 0x2a83840 con 0x208a840 -16054> 2013-04-03 10:38:21.580827 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -16053> 2013-04-03 10:38:21.580835 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -16052> 2013-04-03 10:38:21.580869 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 preprocess_query pg_stats(1 pgs tid 24 v 605) v1 from osd.1 10.252.24.25:6805/3572 -16051> 2013-04-03 10:38:21.580892 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 prepare_update pg_stats(1 pgs tid 24 v 605) v1 from osd.1 10.252.24.25:6805/3572 -16050> 2013-04-03 10:38:21.580898 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 prepare_pg_stats pg_stats(1 pgs tid 24 v 605) v1 from osd.1 -16049> 2013-04-03 10:38:21.580907 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29026 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -16048> 2013-04-03 10:38:21.631090 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29026 encode_pending v 29027 -16047> 2013-04-03 10:38:21.631128 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427339) queue_proposal bl 650 bytes; ctx = 0x2c3d090 -16046> 2013-04-03 10:38:21.631137 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427339) propose_queued 427340 650 bytes -16045> 2013-04-03 10:38:21.632170 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427339 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d500 -16044> 2013-04-03 10:38:21.655884 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113367 ==== paxos(accept lc 427339 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1118670429 0 0) 0x2c4d500 con 0x208a840 -16043> 2013-04-03 10:38:21.656750 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427340 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d780 -16042> 2013-04-03 10:38:21.656778 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427340 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c4da00 -16041> 2013-04-03 10:38:21.656852 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29026 update_from_paxos applying incremental 29027 -16040> 2013-04-03 10:38:21.656867 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 v29027: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -16039> 2013-04-03 10:38:21.656885 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 send_pg_creates to 0 pgs -16038> 2013-04-03 10:38:21.656887 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 update_logger -16037> 2013-04-03 10:38:21.656897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 create_pending v 29028 -16036> 2013-04-03 10:38:21.656905 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29027 _updated_stats for osd.1 10.252.24.25:6805/3572 -16035> 2013-04-03 10:38:21.656910 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 24) v1 tid 17534) v2 -- ?+0 0x2b26b40 con 0x208a840 -16034> 2013-04-03 10:38:21.656920 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 check_osd_map already seen 605 >= 605 -16033> 2013-04-03 10:38:21.656922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 update_logger -16032> 2013-04-03 10:38:21.656932 7fcdffcdb700 0 log [INF] : pgmap v29027: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -16031> 2013-04-03 10:38:21.656934 7fcdffcdb700 10 send_log to self -16030> 2013-04-03 10:38:21.656935 7fcdffcdb700 10 log_queue is 1 last_log 93 sent 92 num 1 unsent 1 sending 1 -16029> 2013-04-03 10:38:21.656936 7fcdffcdb700 10 will send 2013-04-03 10:38:21.656933 mon.0 10.252.24.25:6789/0 93 : [INF] pgmap v29027: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -16028> 2013-04-03 10:38:21.656942 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b5b000 -16027> 2013-04-03 10:38:21.656993 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos -16026> 2013-04-03 10:38:21.657014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 -16025> 2013-04-03 10:38:21.657031 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16024> 2013-04-03 10:38:21.657051 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b5b000 con 0x1bac160 -16023> 2013-04-03 10:38:21.657070 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos -16022> 2013-04-03 10:38:21.657085 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 -16021> 2013-04-03 10:38:21.657093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16020> 2013-04-03 10:38:21.657103 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 preprocess_log log(1 entries) v1 from mon.0 -16019> 2013-04-03 10:38:21.657125 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -16018> 2013-04-03 10:38:21.657137 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 prepare_log log(1 entries) v1 from mon.0 -16017> 2013-04-03 10:38:21.657146 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 logging 2013-04-03 10:38:21.656933 mon.0 10.252.24.25:6789/0 93 : [INF] pgmap v29027: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -16016> 2013-04-03 10:38:21.667832 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113368 ==== paxos(lease_ack lc 427340 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (4038387334 0 0) 0x2c4da00 con 0x208a840 -16015> 2013-04-03 10:38:21.667874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos -16014> 2013-04-03 10:38:21.667890 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394029 update_from_paxos version 394029 summary v 394029 -16013> 2013-04-03 10:38:21.667905 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -16012> 2013-04-03 10:38:21.746764 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113369 ==== forward(pg_stats(1 pgs tid 30443 v 605) v1) to leader v1 ==== 757+0+0 (4171315505 0 0) 0x2a51000 con 0x208a840 -16011> 2013-04-03 10:38:21.746778 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -16010> 2013-04-03 10:38:21.746786 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 -16009> 2013-04-03 10:38:21.746813 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 preprocess_query pg_stats(1 pgs tid 30443 v 605) v1 from osd.3 10.252.24.88:6801/17538 -16008> 2013-04-03 10:38:21.746833 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 prepare_update pg_stats(1 pgs tid 30443 v 605) v1 from osd.3 10.252.24.88:6801/17538 -16007> 2013-04-03 10:38:21.746840 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 prepare_pg_stats pg_stats(1 pgs tid 30443 v 605) v1 from osd.3 -16006> 2013-04-03 10:38:21.746844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29027 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -16005> 2013-04-03 10:38:22.656084 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394029 encode_pending v394030 -16004> 2013-04-03 10:38:22.656117 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427340) queue_proposal bl 443 bytes; ctx = 0x1b91f70 -16003> 2013-04-03 10:38:22.656125 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427340) propose_queued 427341 443 bytes -16002> 2013-04-03 10:38:22.657205 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427340 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d280 -16001> 2013-04-03 10:38:22.657298 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29027 encode_pending v 29028 -16000> 2013-04-03 10:38:22.657343 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427326..427340) queue_proposal bl 654 bytes; ctx = 0x1b91f60 -15999> 2013-04-03 10:38:22.657351 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427326..427340) propose_new_value not active; proposal queued -15998> 2013-04-03 10:38:22.766982 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113370 ==== paxos(accept lc 427340 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3201733622 0 0) 0x2c4d280 con 0x208a840 -15997> 2013-04-03 10:38:22.767279 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427341 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4da00 -15996> 2013-04-03 10:38:22.767312 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427341 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c4d500 -15995> 2013-04-03 10:38:22.767352 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos -15994> 2013-04-03 10:38:22.767379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos version 394030 summary v 394029 -15993> 2013-04-03 10:38:22.767393 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos latest full 393798 -15992> 2013-04-03 10:38:22.767412 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394030 update_from_paxos applying incremental log 394030 2013-04-03 10:38:21.656933 mon.0 10.252.24.25:6789/0 93 : [INF] pgmap v29027: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -15991> 2013-04-03 10:38:22.767457 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 check_subs -15990> 2013-04-03 10:38:22.767488 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 create_pending v 394031 -15989> 2013-04-03 10:38:22.767510 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394030 _updated_log for mon.0 10.252.24.25:6789/0 -15988> 2013-04-03 10:38:22.767523 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 93) v1 -- ?+0 0x2294700 con 0x1bac160 -15987> 2013-04-03 10:38:22.767547 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427341) propose_queued 427342 654 bytes -15986> 2013-04-03 10:38:22.767780 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427341 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fda00 -15985> 2013-04-03 10:38:22.767806 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 93) v1 ==== 0+0+0 (0 0 0) 0x2294700 con 0x1bac160 -15984> 2013-04-03 10:38:22.767813 7fcdffcdb700 10 handle_log_ack log(last 93) v1 -15983> 2013-04-03 10:38:22.767815 7fcdffcdb700 10 logged 2013-04-03 10:38:21.656933 mon.0 10.252.24.25:6789/0 93 : [INF] pgmap v29027: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -15982> 2013-04-03 10:38:22.778871 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113371 ==== paxos(lease_ack lc 427341 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3823435048 0 0) 0x29fda00 con 0x208a840 -15981> 2013-04-03 10:38:22.792103 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113372 ==== paxos(accept lc 427341 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3783942388 0 0) 0x2c4d500 con 0x208a840 -15980> 2013-04-03 10:38:22.793488 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427342 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fda00 -15979> 2013-04-03 10:38:22.793517 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427342 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c4d280 -15978> 2013-04-03 10:38:22.793590 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29027 update_from_paxos applying incremental 29028 -15977> 2013-04-03 10:38:22.793609 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 v29028: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -15976> 2013-04-03 10:38:22.793627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 send_pg_creates to 0 pgs -15975> 2013-04-03 10:38:22.793631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 update_logger -15974> 2013-04-03 10:38:22.793641 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 create_pending v 29029 -15973> 2013-04-03 10:38:22.793650 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29028 _updated_stats for osd.3 10.252.24.88:6801/17538 -15972> 2013-04-03 10:38:22.793655 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30443) v1 tid 17535) v2 -- ?+0 0x24f8b40 con 0x208a840 -15971> 2013-04-03 10:38:22.793665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 check_osd_map already seen 605 >= 605 -15970> 2013-04-03 10:38:22.793667 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 update_logger -15969> 2013-04-03 10:38:22.793676 7fcdffcdb700 0 log [INF] : pgmap v29028: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -15968> 2013-04-03 10:38:22.793678 7fcdffcdb700 10 send_log to self -15967> 2013-04-03 10:38:22.793679 7fcdffcdb700 10 log_queue is 1 last_log 94 sent 93 num 1 unsent 1 sending 1 -15966> 2013-04-03 10:38:22.793680 7fcdffcdb700 10 will send 2013-04-03 10:38:22.793677 mon.0 10.252.24.25:6789/0 94 : [INF] pgmap v29028: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -15965> 2013-04-03 10:38:22.793689 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73800 -15964> 2013-04-03 10:38:22.793735 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos -15963> 2013-04-03 10:38:22.793752 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos version 394030 summary v 394030 -15962> 2013-04-03 10:38:22.793773 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -15961> 2013-04-03 10:38:22.793793 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73800 con 0x1bac160 -15960> 2013-04-03 10:38:22.793815 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos -15959> 2013-04-03 10:38:22.793829 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos version 394030 summary v 394030 -15958> 2013-04-03 10:38:22.793838 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15957> 2013-04-03 10:38:22.793848 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 preprocess_log log(1 entries) v1 from mon.0 -15956> 2013-04-03 10:38:22.793867 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15955> 2013-04-03 10:38:22.793878 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 prepare_log log(1 entries) v1 from mon.0 -15954> 2013-04-03 10:38:22.793888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 logging 2013-04-03 10:38:22.793677 mon.0 10.252.24.25:6789/0 94 : [INF] pgmap v29028: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -15953> 2013-04-03 10:38:22.804070 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113373 ==== paxos(lease_ack lc 427342 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1092751374 0 0) 0x2c4d280 con 0x208a840 -15952> 2013-04-03 10:38:22.804113 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos -15951> 2013-04-03 10:38:22.804129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos version 394030 summary v 394030 -15950> 2013-04-03 10:38:22.804146 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -15949> 2013-04-03 10:38:23.340003 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29028 check_down_pgs -15948> 2013-04-03 10:38:23.340035 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29028 v29028: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -15947> 2013-04-03 10:38:23.340062 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -15946> 2013-04-03 10:38:23.340084 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -15945> 2013-04-03 10:38:23.340249 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -15944> 2013-04-03 10:38:23.340283 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos -15943> 2013-04-03 10:38:23.340301 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394030 update_from_paxos version 394030 summary v 394030 -15942> 2013-04-03 10:38:23.340309 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394030 log -15941> 2013-04-03 10:38:23.340325 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -15940> 2013-04-03 10:38:23.340341 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -15939> 2013-04-03 10:38:23.583647 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 72 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140359 v187) v2 ==== 289+0+0 (1213727937 0 0) 0x2a84080 con 0x2118420 -15938> 2013-04-03 10:38:23.583719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140359 v187) v2 from mds.? 10.252.58.31:6800/10908 -15937> 2013-04-03 10:38:23.583731 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140359 v187) v2 -- ?+0 0x2a51000 con 0x2118420 -15936> 2013-04-03 10:38:23.792282 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394030 encode_pending v394031 -15935> 2013-04-03 10:38:23.792311 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427342) queue_proposal bl 442 bytes; ctx = 0x2c3d0a0 -15934> 2013-04-03 10:38:23.792318 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427342) propose_queued 427343 442 bytes -15933> 2013-04-03 10:38:23.793457 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427342 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d000 -15932> 2013-04-03 10:38:23.828643 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113374 ==== paxos(accept lc 427342 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4133853056 0 0) 0x2c4d000 con 0x208a840 -15931> 2013-04-03 10:38:23.828929 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427343 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d280 -15930> 2013-04-03 10:38:23.828963 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427343 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c4d500 -15929> 2013-04-03 10:38:23.829001 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos -15928> 2013-04-03 10:38:23.829028 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394030 -15927> 2013-04-03 10:38:23.829042 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos latest full 393798 -15926> 2013-04-03 10:38:23.829058 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394031 update_from_paxos applying incremental log 394031 2013-04-03 10:38:22.793677 mon.0 10.252.24.25:6789/0 94 : [INF] pgmap v29028: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -15925> 2013-04-03 10:38:23.829102 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 check_subs -15924> 2013-04-03 10:38:23.829133 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 create_pending v 394032 -15923> 2013-04-03 10:38:23.829155 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394031 _updated_log for mon.0 10.252.24.25:6789/0 -15922> 2013-04-03 10:38:23.829167 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 94) v1 -- ?+0 0x2294700 con 0x1bac160 -15921> 2013-04-03 10:38:23.829216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos -15920> 2013-04-03 10:38:23.829232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394031 -15919> 2013-04-03 10:38:23.829248 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -15918> 2013-04-03 10:38:23.829270 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 94) v1 ==== 0+0+0 (0 0 0) 0x2294700 con 0x1bac160 -15917> 2013-04-03 10:38:23.829278 7fcdffcdb700 10 handle_log_ack log(last 94) v1 -15916> 2013-04-03 10:38:23.829279 7fcdffcdb700 10 logged 2013-04-03 10:38:22.793677 mon.0 10.252.24.25:6789/0 94 : [INF] pgmap v29028: 240 pgs: 240 active+clean; 253 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -15915> 2013-04-03 10:38:23.854062 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113375 ==== paxos(lease_ack lc 427343 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2341821341 0 0) 0x2ab4a00 con 0x208a840 -15914> 2013-04-03 10:38:23.854105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos -15913> 2013-04-03 10:38:23.854121 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394031 -15912> 2013-04-03 10:38:23.854136 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -15911> 2013-04-03 10:38:24.929826 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113376 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 74 v187) v2) to leader v1 ==== 560+0+0 (3359283987 0 0) 0x2a51840 con 0x208a840 -15910> 2013-04-03 10:38:24.929849 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -15909> 2013-04-03 10:38:24.929859 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a512c0 from 10.252.24.88:6789/0 -15908> 2013-04-03 10:38:24.929909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 74 v187) v2 from mds.0 10.252.24.25:6802/3144 -15907> 2013-04-03 10:38:24.929920 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 74 v187) v2 tid 17536) v2 -- ?+0 0x2b26b40 con 0x208a840 -15906> 2013-04-03 10:38:25.383211 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 72 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21730 v187) v2 ==== 289+0+0 (1581511169 0 0) 0x2a83580 con 0x200d420 -15905> 2013-04-03 10:38:25.383283 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21730 v187) v2 from mds.? 10.252.24.88:6800/17213 -15904> 2013-04-03 10:38:25.383295 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21730 v187) v2 -- ?+0 0x2a51840 con 0x200d420 -15903> 2013-04-03 10:38:26.829047 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427343 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c6ec80 -15902> 2013-04-03 10:38:26.829850 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113377 ==== paxos(lease_ack lc 427343 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1883104693 0 0) 0x2c6ec80 con 0x208a840 -15901> 2013-04-03 10:38:26.829922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos -15900> 2013-04-03 10:38:26.829942 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394031 -15899> 2013-04-03 10:38:26.829959 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -15898> 2013-04-03 10:38:27.583776 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 73 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140360 v187) v2 ==== 289+0+0 (3200824747 0 0) 0x2a848c0 con 0x2118420 -15897> 2013-04-03 10:38:27.583848 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140360 v187) v2 from mds.? 10.252.58.31:6800/10908 -15896> 2013-04-03 10:38:27.583861 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140360 v187) v2 -- ?+0 0x2a83580 con 0x2118420 -15895> 2013-04-03 10:38:28.219671 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113378 ==== forward(pg_stats(1 pgs tid 6 v 605) v1) to leader v1 ==== 757+0+0 (3279824119 0 0) 0x2a84080 con 0x208a840 -15894> 2013-04-03 10:38:28.219697 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -15893> 2013-04-03 10:38:28.219707 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 -15892> 2013-04-03 10:38:28.219758 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 preprocess_query pg_stats(1 pgs tid 6 v 605) v1 from osd.2 10.252.24.30:6801/5656 -15891> 2013-04-03 10:38:28.219784 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 prepare_update pg_stats(1 pgs tid 6 v 605) v1 from osd.2 10.252.24.30:6801/5656 -15890> 2013-04-03 10:38:28.219793 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 prepare_pg_stats pg_stats(1 pgs tid 6 v 605) v1 from osd.2 -15889> 2013-04-03 10:38:28.219799 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29028 got osd.2 osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -15888> 2013-04-03 10:38:28.269985 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29028 encode_pending v 29029 -15887> 2013-04-03 10:38:28.270022 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427343) queue_proposal bl 654 bytes; ctx = 0x1b91b00 -15886> 2013-04-03 10:38:28.270032 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427343) propose_queued 427344 654 bytes -15885> 2013-04-03 10:38:28.271276 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427343 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6ea00 -15884> 2013-04-03 10:38:28.298789 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113379 ==== paxos(accept lc 427343 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2670969706 0 0) 0x2c6ea00 con 0x208a840 -15883> 2013-04-03 10:38:28.299136 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427344 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6ec80 -15882> 2013-04-03 10:38:28.299166 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427344 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2ab4a00 -15881> 2013-04-03 10:38:28.299232 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29028 update_from_paxos applying incremental 29029 -15880> 2013-04-03 10:38:28.299248 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29029 v29029: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1388B/s wr, 0op/s -15879> 2013-04-03 10:38:28.299266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29029 send_pg_creates to 0 pgs -15878> 2013-04-03 10:38:28.299271 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29029 update_logger -15877> 2013-04-03 10:38:28.299281 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29029 create_pending v 29030 -15876> 2013-04-03 10:38:28.299289 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29029 _updated_stats for osd.2 10.252.24.30:6801/5656 -15875> 2013-04-03 10:38:28.299295 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 6) v1 tid 17537) v2 -- ?+0 0x24f8b40 con 0x208a840 -15874> 2013-04-03 10:38:28.299307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29029 check_osd_map already seen 605 >= 605 -15873> 2013-04-03 10:38:28.299310 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29029 update_logger -15872> 2013-04-03 10:38:28.299320 7fcdffcdb700 0 log [INF] : pgmap v29029: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1388B/s wr, 0op/s -15871> 2013-04-03 10:38:28.299321 7fcdffcdb700 10 send_log to self -15870> 2013-04-03 10:38:28.299322 7fcdffcdb700 10 log_queue is 1 last_log 95 sent 94 num 1 unsent 1 sending 1 -15869> 2013-04-03 10:38:28.299324 7fcdffcdb700 10 will send 2013-04-03 10:38:28.299321 mon.0 10.252.24.25:6789/0 95 : [INF] pgmap v29029: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1388B/s wr, 0op/s -15868> 2013-04-03 10:38:28.299330 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73800 -15867> 2013-04-03 10:38:28.299380 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos -15866> 2013-04-03 10:38:28.299399 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394031 -15865> 2013-04-03 10:38:28.299417 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -15864> 2013-04-03 10:38:28.299437 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73800 con 0x1bac160 -15863> 2013-04-03 10:38:28.299459 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos -15862> 2013-04-03 10:38:28.299473 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394031 -15861> 2013-04-03 10:38:28.299482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15860> 2013-04-03 10:38:28.299493 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 preprocess_log log(1 entries) v1 from mon.0 -15859> 2013-04-03 10:38:28.299515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15858> 2013-04-03 10:38:28.299527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 prepare_log log(1 entries) v1 from mon.0 -15857> 2013-04-03 10:38:28.299536 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 logging 2013-04-03 10:38:28.299321 mon.0 10.252.24.25:6789/0 95 : [INF] pgmap v29029: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1388B/s wr, 0op/s -15856> 2013-04-03 10:38:28.312045 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113380 ==== paxos(lease_ack lc 427344 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (455338910 0 0) 0x2ab4a00 con 0x208a840 -15855> 2013-04-03 10:38:28.312082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos -15854> 2013-04-03 10:38:28.312098 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394031 -15853> 2013-04-03 10:38:28.312112 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -15852> 2013-04-03 10:38:28.340450 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29029 check_down_pgs -15851> 2013-04-03 10:38:28.340481 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29029 v29029: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1388B/s wr, 0op/s -15850> 2013-04-03 10:38:28.340508 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -15849> 2013-04-03 10:38:28.340529 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -15848> 2013-04-03 10:38:28.340566 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -15847> 2013-04-03 10:38:28.340595 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos -15846> 2013-04-03 10:38:28.340612 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394031 update_from_paxos version 394031 summary v 394031 -15845> 2013-04-03 10:38:28.340620 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394031 log -15844> 2013-04-03 10:38:28.340635 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 update_from_paxos -15843> 2013-04-03 10:38:28.340652 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2060 auth -15842> 2013-04-03 10:38:28.429041 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113381 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (424405211 0 0) 0x2a51580 con 0x208a840 -15841> 2013-04-03 10:38:28.429070 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -15840> 2013-04-03 10:38:28.429077 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -15839> 2013-04-03 10:38:28.429086 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 -15838> 2013-04-03 10:38:28.429094 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( start_reply flags( reply-to(mon.2 10.252.24.88:6789/0) ) ) v1 -- ?+0 0x2a84080 -15837> 2013-04-03 10:38:28.929916 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113382 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 75 v187) v2) to leader v1 ==== 560+0+0 (357539398 0 0) 0x295d580 con 0x208a840 -15836> 2013-04-03 10:38:28.929938 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -15835> 2013-04-03 10:38:28.929948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2145000 from 10.252.24.88:6789/0 -15834> 2013-04-03 10:38:28.929999 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 75 v187) v2 from mds.0 10.252.24.25:6802/3144 -15833> 2013-04-03 10:38:28.930012 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 75 v187) v2 tid 17538) v2 -- ?+0 0x2b26b40 con 0x208a840 -15832> 2013-04-03 10:38:29.298989 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394031 encode_pending v394032 -15831> 2013-04-03 10:38:29.299028 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427344) queue_proposal bl 443 bytes; ctx = 0x1b91fa0 -15830> 2013-04-03 10:38:29.299036 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427344) propose_queued 427345 443 bytes -15829> 2013-04-03 10:38:29.300126 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427344 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6e780 -15828> 2013-04-03 10:38:29.326794 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113383 ==== paxos(accept lc 427344 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (422245245 0 0) 0x2c6e780 con 0x208a840 -15827> 2013-04-03 10:38:29.327115 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427345 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4a00 -15826> 2013-04-03 10:38:29.327148 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427345 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c6ea00 -15825> 2013-04-03 10:38:29.327188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos -15824> 2013-04-03 10:38:29.327215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394031 -15823> 2013-04-03 10:38:29.327230 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos latest full 393798 -15822> 2013-04-03 10:38:29.327247 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos applying incremental log 394032 2013-04-03 10:38:28.299321 mon.0 10.252.24.25:6789/0 95 : [INF] pgmap v29029: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1388B/s wr, 0op/s -15821> 2013-04-03 10:38:29.327294 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 check_subs -15820> 2013-04-03 10:38:29.327326 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 create_pending v 394033 -15819> 2013-04-03 10:38:29.327349 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 _updated_log for mon.0 10.252.24.25:6789/0 -15818> 2013-04-03 10:38:29.327362 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 95) v1 -- ?+0 0x2294380 con 0x1bac160 -15817> 2013-04-03 10:38:29.327413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos -15816> 2013-04-03 10:38:29.327429 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 -15815> 2013-04-03 10:38:29.327451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15814> 2013-04-03 10:38:29.327474 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 95) v1 ==== 0+0+0 (0 0 0) 0x2294380 con 0x1bac160 -15813> 2013-04-03 10:38:29.327482 7fcdffcdb700 10 handle_log_ack log(last 95) v1 -15812> 2013-04-03 10:38:29.327483 7fcdffcdb700 10 logged 2013-04-03 10:38:28.299321 mon.0 10.252.24.25:6789/0 95 : [INF] pgmap v29029: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1388B/s wr, 0op/s -15811> 2013-04-03 10:38:29.352006 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113384 ==== paxos(lease_ack lc 427345 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2658891720 0 0) 0x2c6ea00 con 0x208a840 -15810> 2013-04-03 10:38:29.352047 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos -15809> 2013-04-03 10:38:29.352063 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 -15808> 2013-04-03 10:38:29.352078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15807> 2013-04-03 10:38:29.383284 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 73 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21731 v187) v2 ==== 289+0+0 (2100120956 0 0) 0x2995840 con 0x200d420 -15806> 2013-04-03 10:38:29.383337 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21731 v187) v2 from mds.? 10.252.24.88:6800/17213 -15805> 2013-04-03 10:38:29.383349 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21731 v187) v2 -- ?+0 0x295d580 con 0x200d420 -15804> 2013-04-03 10:38:31.583922 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 74 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140361 v187) v2 ==== 289+0+0 (2644904662 0 0) 0x29c58c0 con 0x2118420 -15803> 2013-04-03 10:38:31.583995 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140361 v187) v2 from mds.? 10.252.58.31:6800/10908 -15802> 2013-04-03 10:38:31.584008 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140361 v187) v2 -- ?+0 0x2995840 con 0x2118420 -15801> 2013-04-03 10:38:32.327225 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427345 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c6e500 -15800> 2013-04-03 10:38:32.328039 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113385 ==== paxos(lease_ack lc 427345 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1287595570 0 0) 0x2c6e500 con 0x208a840 -15799> 2013-04-03 10:38:32.328106 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos -15798> 2013-04-03 10:38:32.328126 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 -15797> 2013-04-03 10:38:32.328143 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15796> 2013-04-03 10:38:32.930027 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113386 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 76 v187) v2) to leader v1 ==== 560+0+0 (2250720274 0 0) 0x2a51580 con 0x208a840 -15795> 2013-04-03 10:38:32.930050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -15794> 2013-04-03 10:38:32.930061 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29c5600 from 10.252.24.88:6789/0 -15793> 2013-04-03 10:38:32.930106 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 76 v187) v2 from mds.0 10.252.24.25:6802/3144 -15792> 2013-04-03 10:38:32.930119 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 76 v187) v2 tid 17539) v2 -- ?+0 0x24f8b40 con 0x208a840 -15791> 2013-04-03 10:38:33.340794 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29029 v29029: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1388B/s wr, 0op/s -15790> 2013-04-03 10:38:33.340837 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -15789> 2013-04-03 10:38:33.340863 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -15788> 2013-04-03 10:38:33.340908 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -15787> 2013-04-03 10:38:33.340939 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos -15786> 2013-04-03 10:38:33.340957 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 -15785> 2013-04-03 10:38:33.340967 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 log -15784> 2013-04-03 10:38:33.340984 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15783> 2013-04-03 10:38:33.341002 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -15782> 2013-04-03 10:38:33.383408 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 74 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21732 v187) v2 ==== 289+0+0 (2502193423 0 0) 0x2995dc0 con 0x200d420 -15781> 2013-04-03 10:38:33.383454 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21732 v187) v2 from mds.? 10.252.24.88:6800/17213 -15780> 2013-04-03 10:38:33.383466 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21732 v187) v2 -- ?+0 0x2a51580 con 0x200d420 -15779> 2013-04-03 10:38:35.327294 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427345 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c6e280 -15778> 2013-04-03 10:38:35.328097 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113387 ==== paxos(lease_ack lc 427345 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3074169982 0 0) 0x2c6e280 con 0x208a840 -15777> 2013-04-03 10:38:35.328164 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos -15776> 2013-04-03 10:38:35.328184 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 -15775> 2013-04-03 10:38:35.328201 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15774> 2013-04-03 10:38:35.584012 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 75 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140362 v187) v2 ==== 289+0+0 (4162177873 0 0) 0x29c42c0 con 0x2118420 -15773> 2013-04-03 10:38:35.584663 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140362 v187) v2 from mds.? 10.252.58.31:6800/10908 -15772> 2013-04-03 10:38:35.584683 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140362 v187) v2 -- ?+0 0x2995dc0 con 0x2118420 -15771> 2013-04-03 10:38:36.582518 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113388 ==== forward(pg_stats(1 pgs tid 25 v 605) v1) to leader v1 ==== 753+0+0 (3356367937 0 0) 0x29c58c0 con 0x208a840 -15770> 2013-04-03 10:38:36.582537 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -15769> 2013-04-03 10:38:36.582548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -15768> 2013-04-03 10:38:36.582599 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29029 preprocess_query pg_stats(1 pgs tid 25 v 605) v1 from osd.1 10.252.24.25:6805/3572 -15767> 2013-04-03 10:38:36.582623 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29029 prepare_update pg_stats(1 pgs tid 25 v 605) v1 from osd.1 10.252.24.25:6805/3572 -15766> 2013-04-03 10:38:36.582633 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29029 prepare_pg_stats pg_stats(1 pgs tid 25 v 605) v1 from osd.1 -15765> 2013-04-03 10:38:36.582640 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29029 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -15764> 2013-04-03 10:38:36.632832 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29029 encode_pending v 29030 -15763> 2013-04-03 10:38:36.632872 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427345) queue_proposal bl 650 bytes; ctx = 0x2c3d060 -15762> 2013-04-03 10:38:36.632881 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427345) propose_queued 427346 650 bytes -15761> 2013-04-03 10:38:36.634205 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427345 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6e000 -15760> 2013-04-03 10:38:36.657654 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113389 ==== paxos(accept lc 427345 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4231617037 0 0) 0x2c6e000 con 0x208a840 -15759> 2013-04-03 10:38:36.658002 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427346 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6e280 -15758> 2013-04-03 10:38:36.658032 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427346 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c6e500 -15757> 2013-04-03 10:38:36.658107 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29029 update_from_paxos applying incremental 29030 -15756> 2013-04-03 10:38:36.658123 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 v29030: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 879B/s wr, 0op/s -15755> 2013-04-03 10:38:36.658144 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 send_pg_creates to 0 pgs -15754> 2013-04-03 10:38:36.658147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 update_logger -15753> 2013-04-03 10:38:36.658157 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 create_pending v 29031 -15752> 2013-04-03 10:38:36.658166 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 _updated_stats for osd.1 10.252.24.25:6805/3572 -15751> 2013-04-03 10:38:36.658172 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 25) v1 tid 17540) v2 -- ?+0 0x2b26b40 con 0x208a840 -15750> 2013-04-03 10:38:36.658182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 check_osd_map already seen 605 >= 605 -15749> 2013-04-03 10:38:36.658185 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 update_logger -15748> 2013-04-03 10:38:36.658196 7fcdffcdb700 0 log [INF] : pgmap v29030: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 879B/s wr, 0op/s -15747> 2013-04-03 10:38:36.658197 7fcdffcdb700 10 send_log to self -15746> 2013-04-03 10:38:36.658198 7fcdffcdb700 10 log_queue is 1 last_log 96 sent 95 num 1 unsent 1 sending 1 -15745> 2013-04-03 10:38:36.658200 7fcdffcdb700 10 will send 2013-04-03 10:38:36.658197 mon.0 10.252.24.25:6789/0 96 : [INF] pgmap v29030: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 879B/s wr, 0op/s -15744> 2013-04-03 10:38:36.658206 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73800 -15743> 2013-04-03 10:38:36.658263 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos -15742> 2013-04-03 10:38:36.658282 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 -15741> 2013-04-03 10:38:36.658301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15740> 2013-04-03 10:38:36.658325 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73800 con 0x1bac160 -15739> 2013-04-03 10:38:36.658348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos -15738> 2013-04-03 10:38:36.658363 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 -15737> 2013-04-03 10:38:36.658372 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15736> 2013-04-03 10:38:36.658384 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 preprocess_log log(1 entries) v1 from mon.0 -15735> 2013-04-03 10:38:36.658409 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15734> 2013-04-03 10:38:36.658423 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 prepare_log log(1 entries) v1 from mon.0 -15733> 2013-04-03 10:38:36.658433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 logging 2013-04-03 10:38:36.658197 mon.0 10.252.24.25:6789/0 96 : [INF] pgmap v29030: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 879B/s wr, 0op/s -15732> 2013-04-03 10:38:36.687629 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113390 ==== paxos(lease_ack lc 427346 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3246885924 0 0) 0x2c6e500 con 0x208a840 -15731> 2013-04-03 10:38:36.687666 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos -15730> 2013-04-03 10:38:36.687682 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 update_from_paxos version 394032 summary v 394032 -15729> 2013-04-03 10:38:36.687697 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15728> 2013-04-03 10:38:36.930115 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113391 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 77 v187) v2) to leader v1 ==== 560+0+0 (1861161854 0 0) 0x29c4b00 con 0x208a840 -15727> 2013-04-03 10:38:36.930129 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -15726> 2013-04-03 10:38:36.930139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29c5080 from 10.252.24.88:6789/0 -15725> 2013-04-03 10:38:36.930171 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 77 v187) v2 from mds.0 10.252.24.25:6802/3144 -15724> 2013-04-03 10:38:36.930183 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 77 v187) v2 tid 17541) v2 -- ?+0 0x24f8b40 con 0x208a840 -15723> 2013-04-03 10:38:37.383481 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 75 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21733 v187) v2 ==== 289+0+0 (3058355826 0 0) 0x273b8c0 con 0x200d420 -15722> 2013-04-03 10:38:37.383558 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21733 v187) v2 from mds.? 10.252.24.88:6800/17213 -15721> 2013-04-03 10:38:37.383572 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21733 v187) v2 -- ?+0 0x29c4b00 con 0x200d420 -15720> 2013-04-03 10:38:37.657854 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394032 encode_pending v394033 -15719> 2013-04-03 10:38:37.657889 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427346) queue_proposal bl 442 bytes; ctx = 0x1b91f40 -15718> 2013-04-03 10:38:37.657896 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427346) propose_queued 427347 442 bytes -15717> 2013-04-03 10:38:37.658818 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427346 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d500 -15716> 2013-04-03 10:38:37.710692 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113392 ==== paxos(accept lc 427346 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2504897809 0 0) 0x2c4d500 con 0x208a840 -15715> 2013-04-03 10:38:37.719259 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427347 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6e500 -15714> 2013-04-03 10:38:37.719295 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427347 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c6e000 -15713> 2013-04-03 10:38:37.719340 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos -15712> 2013-04-03 10:38:37.719367 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394032 -15711> 2013-04-03 10:38:37.719382 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos latest full 393798 -15710> 2013-04-03 10:38:37.719399 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos applying incremental log 394033 2013-04-03 10:38:36.658197 mon.0 10.252.24.25:6789/0 96 : [INF] pgmap v29030: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 879B/s wr, 0op/s -15709> 2013-04-03 10:38:37.719457 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 check_subs -15708> 2013-04-03 10:38:37.719490 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 create_pending v 394034 -15707> 2013-04-03 10:38:37.719513 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 _updated_log for mon.0 10.252.24.25:6789/0 -15706> 2013-04-03 10:38:37.719527 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 96) v1 -- ?+0 0x22941c0 con 0x1bac160 -15705> 2013-04-03 10:38:37.719578 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos -15704> 2013-04-03 10:38:37.719593 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394033 -15703> 2013-04-03 10:38:37.719612 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15702> 2013-04-03 10:38:37.719633 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 96) v1 ==== 0+0+0 (0 0 0) 0x22941c0 con 0x1bac160 -15701> 2013-04-03 10:38:37.719643 7fcdffcdb700 10 handle_log_ack log(last 96) v1 -15700> 2013-04-03 10:38:37.719646 7fcdffcdb700 10 logged 2013-04-03 10:38:36.658197 mon.0 10.252.24.25:6789/0 96 : [INF] pgmap v29030: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 879B/s wr, 0op/s -15699> 2013-04-03 10:38:37.732351 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113393 ==== paxos(lease_ack lc 427347 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (803122881 0 0) 0x2c6e000 con 0x208a840 -15698> 2013-04-03 10:38:37.732385 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos -15697> 2013-04-03 10:38:37.732402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394033 -15696> 2013-04-03 10:38:37.732417 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15695> 2013-04-03 10:38:38.341144 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 check_down_pgs -15694> 2013-04-03 10:38:38.341176 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 v29030: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 879B/s wr, 0op/s -15693> 2013-04-03 10:38:38.341203 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -15692> 2013-04-03 10:38:38.341228 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -15691> 2013-04-03 10:38:38.341265 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -15690> 2013-04-03 10:38:38.341297 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos -15689> 2013-04-03 10:38:38.341328 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394033 -15688> 2013-04-03 10:38:38.341338 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 log -15687> 2013-04-03 10:38:38.341356 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15686> 2013-04-03 10:38:38.341375 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -15685> 2013-04-03 10:38:39.584143 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 76 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140363 v187) v2 ==== 289+0+0 (3682300972 0 0) 0x29c5b80 con 0x2118420 -15684> 2013-04-03 10:38:39.584218 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140363 v187) v2 from mds.? 10.252.58.31:6800/10908 -15683> 2013-04-03 10:38:39.584231 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140363 v187) v2 -- ?+0 0x273b8c0 con 0x2118420 -15682> 2013-04-03 10:38:40.719370 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427347 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c4d280 -15681> 2013-04-03 10:38:40.720241 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113394 ==== paxos(lease_ack lc 427347 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1784513823 0 0) 0x2c4d280 con 0x208a840 -15680> 2013-04-03 10:38:40.720307 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos -15679> 2013-04-03 10:38:40.720327 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394033 -15678> 2013-04-03 10:38:40.720345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15677> 2013-04-03 10:38:40.930279 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113395 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 78 v187) v2) to leader v1 ==== 560+0+0 (1966436731 0 0) 0x29c58c0 con 0x208a840 -15676> 2013-04-03 10:38:40.930293 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -15675> 2013-04-03 10:38:40.930302 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29c4840 from 10.252.24.88:6789/0 -15674> 2013-04-03 10:38:40.930334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 78 v187) v2 from mds.0 10.252.24.25:6802/3144 -15673> 2013-04-03 10:38:40.930346 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 78 v187) v2 tid 17542) v2 -- ?+0 0x2b26b40 con 0x208a840 -15672> 2013-04-03 10:38:41.383635 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 76 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21734 v187) v2 ==== 289+0+0 (3556315125 0 0) 0x2a83dc0 con 0x200d420 -15671> 2013-04-03 10:38:41.383715 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21734 v187) v2 from mds.? 10.252.24.88:6800/17213 -15670> 2013-04-03 10:38:41.383730 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21734 v187) v2 -- ?+0 0x29c58c0 con 0x200d420 -15669> 2013-04-03 10:38:41.583368 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113396 ==== forward(pg_stats(1 pgs tid 26 v 605) v1) to leader v1 ==== 753+0+0 (1585995631 0 0) 0x29c5b80 con 0x208a840 -15668> 2013-04-03 10:38:41.583396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -15667> 2013-04-03 10:38:41.583407 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 -15666> 2013-04-03 10:38:41.583457 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 preprocess_query pg_stats(1 pgs tid 26 v 605) v1 from osd.1 10.252.24.25:6805/3572 -15665> 2013-04-03 10:38:41.583481 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 prepare_update pg_stats(1 pgs tid 26 v 605) v1 from osd.1 10.252.24.25:6805/3572 -15664> 2013-04-03 10:38:41.583488 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 prepare_pg_stats pg_stats(1 pgs tid 26 v 605) v1 from osd.1 -15663> 2013-04-03 10:38:41.583494 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -15662> 2013-04-03 10:38:41.633681 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 encode_pending v 29031 -15661> 2013-04-03 10:38:41.633728 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427347) queue_proposal bl 650 bytes; ctx = 0x2c3d050 -15660> 2013-04-03 10:38:41.633738 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427347) propose_queued 427348 650 bytes -15659> 2013-04-03 10:38:41.635809 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427347 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4da00 -15658> 2013-04-03 10:38:41.661141 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113397 ==== paxos(accept lc 427347 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1393153776 0 0) 0x2c4da00 con 0x208a840 -15657> 2013-04-03 10:38:41.661497 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427348 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d280 -15656> 2013-04-03 10:38:41.661525 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427348 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c6e000 -15655> 2013-04-03 10:38:41.662386 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29030 update_from_paxos applying incremental 29031 -15654> 2013-04-03 10:38:41.662403 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 v29031: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 383B/s wr, 0op/s -15653> 2013-04-03 10:38:41.662424 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 send_pg_creates to 0 pgs -15652> 2013-04-03 10:38:41.662428 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 update_logger -15651> 2013-04-03 10:38:41.662439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 create_pending v 29032 -15650> 2013-04-03 10:38:41.662448 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 _updated_stats for osd.1 10.252.24.25:6805/3572 -15649> 2013-04-03 10:38:41.662456 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 26) v1 tid 17543) v2 -- ?+0 0x24f8b40 con 0x208a840 -15648> 2013-04-03 10:38:41.662470 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 check_osd_map already seen 605 >= 605 -15647> 2013-04-03 10:38:41.662474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 update_logger -15646> 2013-04-03 10:38:41.662484 7fcdffcdb700 0 log [INF] : pgmap v29031: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 383B/s wr, 0op/s -15645> 2013-04-03 10:38:41.662486 7fcdffcdb700 10 send_log to self -15644> 2013-04-03 10:38:41.662487 7fcdffcdb700 10 log_queue is 1 last_log 97 sent 96 num 1 unsent 1 sending 1 -15643> 2013-04-03 10:38:41.662488 7fcdffcdb700 10 will send 2013-04-03 10:38:41.662485 mon.0 10.252.24.25:6789/0 97 : [INF] pgmap v29031: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 383B/s wr, 0op/s -15642> 2013-04-03 10:38:41.662495 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73800 -15641> 2013-04-03 10:38:41.662727 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos -15640> 2013-04-03 10:38:41.662749 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394033 -15639> 2013-04-03 10:38:41.662768 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15638> 2013-04-03 10:38:41.662789 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73800 con 0x1bac160 -15637> 2013-04-03 10:38:41.662812 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos -15636> 2013-04-03 10:38:41.662827 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394033 -15635> 2013-04-03 10:38:41.662836 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15634> 2013-04-03 10:38:41.662848 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 preprocess_log log(1 entries) v1 from mon.0 -15633> 2013-04-03 10:38:41.662873 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15632> 2013-04-03 10:38:41.662886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 prepare_log log(1 entries) v1 from mon.0 -15631> 2013-04-03 10:38:41.662898 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 logging 2013-04-03 10:38:41.662485 mon.0 10.252.24.25:6789/0 97 : [INF] pgmap v29031: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 383B/s wr, 0op/s -15630> 2013-04-03 10:38:41.674400 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113398 ==== paxos(lease_ack lc 427348 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (601794477 0 0) 0x2c6e000 con 0x208a840 -15629> 2013-04-03 10:38:41.674437 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos -15628> 2013-04-03 10:38:41.674453 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 update_from_paxos version 394033 summary v 394033 -15627> 2013-04-03 10:38:41.674468 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15626> 2013-04-03 10:38:41.748920 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113399 ==== forward(pg_stats(1 pgs tid 30444 v 605) v1) to leader v1 ==== 757+0+0 (2857267892 0 0) 0x29c4dc0 con 0x208a840 -15625> 2013-04-03 10:38:41.748933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -15624> 2013-04-03 10:38:41.748941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -15623> 2013-04-03 10:38:41.748970 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 preprocess_query pg_stats(1 pgs tid 30444 v 605) v1 from osd.3 10.252.24.88:6801/17538 -15622> 2013-04-03 10:38:41.748991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 prepare_update pg_stats(1 pgs tid 30444 v 605) v1 from osd.3 10.252.24.88:6801/17538 -15621> 2013-04-03 10:38:41.748997 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 prepare_pg_stats pg_stats(1 pgs tid 30444 v 605) v1 from osd.3 -15620> 2013-04-03 10:38:41.749003 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 got osd.3 osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -15619> 2013-04-03 10:38:42.661351 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394033 encode_pending v394034 -15618> 2013-04-03 10:38:42.661382 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427348) queue_proposal bl 442 bytes; ctx = 0x2c3d080 -15617> 2013-04-03 10:38:42.661390 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427348) propose_queued 427349 442 bytes -15616> 2013-04-03 10:38:42.662449 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427348 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d780 -15615> 2013-04-03 10:38:42.662542 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 encode_pending v 29032 -15614> 2013-04-03 10:38:42.662579 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427326..427348) queue_proposal bl 654 bytes; ctx = 0x2c3d170 -15613> 2013-04-03 10:38:42.662586 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427326..427348) propose_new_value not active; proposal queued -15612> 2013-04-03 10:38:42.767593 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113400 ==== paxos(accept lc 427348 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2715776170 0 0) 0x2c4d780 con 0x208a840 -15611> 2013-04-03 10:38:42.769167 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427349 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6e000 -15610> 2013-04-03 10:38:42.769207 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427349 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c4da00 -15609> 2013-04-03 10:38:42.769277 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos -15608> 2013-04-03 10:38:42.769329 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos version 394034 summary v 394033 -15607> 2013-04-03 10:38:42.769344 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos latest full 393798 -15606> 2013-04-03 10:38:42.769363 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos applying incremental log 394034 2013-04-03 10:38:41.662485 mon.0 10.252.24.25:6789/0 97 : [INF] pgmap v29031: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 383B/s wr, 0op/s -15605> 2013-04-03 10:38:42.769450 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 check_subs -15604> 2013-04-03 10:38:42.769485 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 create_pending v 394035 -15603> 2013-04-03 10:38:42.769508 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 _updated_log for mon.0 10.252.24.25:6789/0 -15602> 2013-04-03 10:38:42.769522 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 97) v1 -- ?+0 0x28a58c0 con 0x1bac160 -15601> 2013-04-03 10:38:42.769549 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427349) propose_queued 427350 654 bytes -15600> 2013-04-03 10:38:42.769802 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427349 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d500 -15599> 2013-04-03 10:38:42.769836 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 97) v1 ==== 0+0+0 (0 0 0) 0x28a58c0 con 0x1bac160 -15598> 2013-04-03 10:38:42.769844 7fcdffcdb700 10 handle_log_ack log(last 97) v1 -15597> 2013-04-03 10:38:42.769847 7fcdffcdb700 10 logged 2013-04-03 10:38:41.662485 mon.0 10.252.24.25:6789/0 97 : [INF] pgmap v29031: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 383B/s wr, 0op/s -15596> 2013-04-03 10:38:42.779788 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113401 ==== paxos(lease_ack lc 427349 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2250231128 0 0) 0x2c4d500 con 0x208a840 -15595> 2013-04-03 10:38:42.805669 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113402 ==== paxos(accept lc 427349 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2199799608 0 0) 0x2c4da00 con 0x208a840 -15594> 2013-04-03 10:38:42.806017 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427350 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d500 -15593> 2013-04-03 10:38:42.806049 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427350 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c4d780 -15592> 2013-04-03 10:38:42.806132 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29031 update_from_paxos applying incremental 29032 -15591> 2013-04-03 10:38:42.806152 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 v29032: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 339B/s wr, 0op/s -15590> 2013-04-03 10:38:42.806172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 send_pg_creates to 0 pgs -15589> 2013-04-03 10:38:42.806176 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 update_logger -15588> 2013-04-03 10:38:42.806187 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 create_pending v 29033 -15587> 2013-04-03 10:38:42.806198 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 _updated_stats for osd.3 10.252.24.88:6801/17538 -15586> 2013-04-03 10:38:42.806206 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30444) v1 tid 17544) v2 -- ?+0 0x2b26b40 con 0x208a840 -15585> 2013-04-03 10:38:42.806215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 check_osd_map already seen 605 >= 605 -15584> 2013-04-03 10:38:42.806221 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 update_logger -15583> 2013-04-03 10:38:42.806231 7fcdffcdb700 0 log [INF] : pgmap v29032: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 339B/s wr, 0op/s -15582> 2013-04-03 10:38:42.806233 7fcdffcdb700 10 send_log to self -15581> 2013-04-03 10:38:42.806234 7fcdffcdb700 10 log_queue is 1 last_log 98 sent 97 num 1 unsent 1 sending 1 -15580> 2013-04-03 10:38:42.806235 7fcdffcdb700 10 will send 2013-04-03 10:38:42.806232 mon.0 10.252.24.25:6789/0 98 : [INF] pgmap v29032: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 339B/s wr, 0op/s -15579> 2013-04-03 10:38:42.806241 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73000 -15578> 2013-04-03 10:38:42.806292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos -15577> 2013-04-03 10:38:42.806312 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos version 394034 summary v 394034 -15576> 2013-04-03 10:38:42.806331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15575> 2013-04-03 10:38:42.806352 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73000 con 0x1bac160 -15574> 2013-04-03 10:38:42.806373 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos -15573> 2013-04-03 10:38:42.806389 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos version 394034 summary v 394034 -15572> 2013-04-03 10:38:42.806399 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15571> 2013-04-03 10:38:42.806411 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 preprocess_log log(1 entries) v1 from mon.0 -15570> 2013-04-03 10:38:42.806435 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15569> 2013-04-03 10:38:42.806449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 prepare_log log(1 entries) v1 from mon.0 -15568> 2013-04-03 10:38:42.806460 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 logging 2013-04-03 10:38:42.806232 mon.0 10.252.24.25:6789/0 98 : [INF] pgmap v29032: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 339B/s wr, 0op/s -15567> 2013-04-03 10:38:42.819350 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113403 ==== paxos(lease_ack lc 427350 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (967699585 0 0) 0x2c4d780 con 0x208a840 -15566> 2013-04-03 10:38:42.819391 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos -15565> 2013-04-03 10:38:42.819410 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos version 394034 summary v 394034 -15564> 2013-04-03 10:38:42.819427 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15563> 2013-04-03 10:38:43.222043 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113404 ==== forward(pg_stats(1 pgs tid 7 v 605) v1) to leader v1 ==== 757+0+0 (1326841825 0 0) 0x273b600 con 0x208a840 -15562> 2013-04-03 10:38:43.222069 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -15561> 2013-04-03 10:38:43.222081 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 -15560> 2013-04-03 10:38:43.222133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 preprocess_query pg_stats(1 pgs tid 7 v 605) v1 from osd.2 10.252.24.30:6801/5656 -15559> 2013-04-03 10:38:43.222158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 prepare_update pg_stats(1 pgs tid 7 v 605) v1 from osd.2 10.252.24.30:6801/5656 -15558> 2013-04-03 10:38:43.222165 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 prepare_pg_stats pg_stats(1 pgs tid 7 v 605) v1 from osd.2 -15557> 2013-04-03 10:38:43.222172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1174 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -15556> 2013-04-03 10:38:43.341505 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 check_down_pgs -15555> 2013-04-03 10:38:43.341538 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 v29032: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 339B/s wr, 0op/s -15554> 2013-04-03 10:38:43.341567 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -15553> 2013-04-03 10:38:43.341589 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -15552> 2013-04-03 10:38:43.341625 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -15551> 2013-04-03 10:38:43.341659 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos -15550> 2013-04-03 10:38:43.341679 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 update_from_paxos version 394034 summary v 394034 -15549> 2013-04-03 10:38:43.341688 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 log -15548> 2013-04-03 10:38:43.341705 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15547> 2013-04-03 10:38:43.341724 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -15546> 2013-04-03 10:38:43.584260 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 77 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140364 v187) v2 ==== 289+0+0 (863127647 0 0) 0x2996b80 con 0x2118420 -15545> 2013-04-03 10:38:43.584306 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140364 v187) v2 from mds.? 10.252.58.31:6800/10908 -15544> 2013-04-03 10:38:43.584318 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140364 v187) v2 -- ?+0 0x273b600 con 0x2118420 -15543> 2013-04-03 10:38:43.805870 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394034 encode_pending v394035 -15542> 2013-04-03 10:38:43.805908 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427350) queue_proposal bl 442 bytes; ctx = 0x1b91f30 -15541> 2013-04-03 10:38:43.805916 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427350) propose_queued 427351 442 bytes -15540> 2013-04-03 10:38:43.807023 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427350 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fda00 -15539> 2013-04-03 10:38:43.807120 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 encode_pending v 29033 -15538> 2013-04-03 10:38:43.807154 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427326..427350) queue_proposal bl 654 bytes; ctx = 0x2c3d160 -15537> 2013-04-03 10:38:43.807161 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427326..427350) propose_new_value not active; proposal queued -15536> 2013-04-03 10:38:43.829944 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113405 ==== paxos(accept lc 427350 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3562952606 0 0) 0x29fda00 con 0x208a840 -15535> 2013-04-03 10:38:43.839384 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427351 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d780 -15534> 2013-04-03 10:38:43.839420 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427351 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c4da00 -15533> 2013-04-03 10:38:43.839462 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos -15532> 2013-04-03 10:38:43.839488 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos version 394035 summary v 394034 -15531> 2013-04-03 10:38:43.839504 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos latest full 393798 -15530> 2013-04-03 10:38:43.839523 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos applying incremental log 394035 2013-04-03 10:38:42.806232 mon.0 10.252.24.25:6789/0 98 : [INF] pgmap v29032: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 339B/s wr, 0op/s -15529> 2013-04-03 10:38:43.839574 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 check_subs -15528> 2013-04-03 10:38:43.839606 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 create_pending v 394036 -15527> 2013-04-03 10:38:43.839629 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 _updated_log for mon.0 10.252.24.25:6789/0 -15526> 2013-04-03 10:38:43.839642 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 98) v1 -- ?+0 0x28a5700 con 0x1bac160 -15525> 2013-04-03 10:38:43.839666 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427351) propose_queued 427352 654 bytes -15524> 2013-04-03 10:38:43.839871 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427351 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6ea00 -15523> 2013-04-03 10:38:43.839905 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 98) v1 ==== 0+0+0 (0 0 0) 0x28a5700 con 0x1bac160 -15522> 2013-04-03 10:38:43.839923 7fcdffcdb700 10 handle_log_ack log(last 98) v1 -15521> 2013-04-03 10:38:43.839924 7fcdffcdb700 10 logged 2013-04-03 10:38:42.806232 mon.0 10.252.24.25:6789/0 98 : [INF] pgmap v29032: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 339B/s wr, 0op/s -15520> 2013-04-03 10:38:43.850657 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113406 ==== paxos(lease_ack lc 427351 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1001874185 0 0) 0x2c6ea00 con 0x208a840 -15519> 2013-04-03 10:38:43.864247 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113407 ==== paxos(accept lc 427351 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (896239081 0 0) 0x2c4da00 con 0x208a840 -15518> 2013-04-03 10:38:43.865521 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427352 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6ea00 -15517> 2013-04-03 10:38:43.865554 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427352 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29fda00 -15516> 2013-04-03 10:38:43.865629 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29032 update_from_paxos applying incremental 29033 -15515> 2013-04-03 10:38:43.865648 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 v29033: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 8009B/s wr, 0op/s -15514> 2013-04-03 10:38:43.865668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 send_pg_creates to 0 pgs -15513> 2013-04-03 10:38:43.865671 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 update_logger -15512> 2013-04-03 10:38:43.865681 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 create_pending v 29034 -15511> 2013-04-03 10:38:43.865691 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 _updated_stats for osd.2 10.252.24.30:6801/5656 -15510> 2013-04-03 10:38:43.865700 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 7) v1 tid 17545) v2 -- ?+0 0x24f8b40 con 0x208a840 -15509> 2013-04-03 10:38:43.865711 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 check_osd_map already seen 605 >= 605 -15508> 2013-04-03 10:38:43.865714 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 update_logger -15507> 2013-04-03 10:38:43.865725 7fcdffcdb700 0 log [INF] : pgmap v29033: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 8009B/s wr, 0op/s -15506> 2013-04-03 10:38:43.865727 7fcdffcdb700 10 send_log to self -15505> 2013-04-03 10:38:43.865727 7fcdffcdb700 10 log_queue is 1 last_log 99 sent 98 num 1 unsent 1 sending 1 -15504> 2013-04-03 10:38:43.865729 7fcdffcdb700 10 will send 2013-04-03 10:38:43.865726 mon.0 10.252.24.25:6789/0 99 : [INF] pgmap v29033: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 8009B/s wr, 0op/s -15503> 2013-04-03 10:38:43.865735 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74e00 -15502> 2013-04-03 10:38:43.865782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos -15501> 2013-04-03 10:38:43.865799 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos version 394035 summary v 394035 -15500> 2013-04-03 10:38:43.865819 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15499> 2013-04-03 10:38:43.865839 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74e00 con 0x1bac160 -15498> 2013-04-03 10:38:43.865861 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos -15497> 2013-04-03 10:38:43.865876 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos version 394035 summary v 394035 -15496> 2013-04-03 10:38:43.865886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15495> 2013-04-03 10:38:43.865899 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 preprocess_log log(1 entries) v1 from mon.0 -15494> 2013-04-03 10:38:43.865918 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15493> 2013-04-03 10:38:43.865931 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 prepare_log log(1 entries) v1 from mon.0 -15492> 2013-04-03 10:38:43.865941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 logging 2013-04-03 10:38:43.865726 mon.0 10.252.24.25:6789/0 99 : [INF] pgmap v29033: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 8009B/s wr, 0op/s -15491> 2013-04-03 10:38:43.890379 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113408 ==== paxos(lease_ack lc 427352 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (199037036 0 0) 0x29fda00 con 0x208a840 -15490> 2013-04-03 10:38:43.890421 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos -15489> 2013-04-03 10:38:43.890439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 update_from_paxos version 394035 summary v 394035 -15488> 2013-04-03 10:38:43.890455 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15487> 2013-04-03 10:38:44.864415 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394035 encode_pending v394036 -15486> 2013-04-03 10:38:44.864442 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427352) queue_proposal bl 443 bytes; ctx = 0x2c3d040 -15485> 2013-04-03 10:38:44.864450 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427352) propose_queued 427353 443 bytes -15484> 2013-04-03 10:38:44.865771 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427352 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd280 -15483> 2013-04-03 10:38:44.890719 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113409 ==== paxos(accept lc 427352 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1926504573 0 0) 0x29fd280 con 0x208a840 -15482> 2013-04-03 10:38:44.897739 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427353 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fda00 -15481> 2013-04-03 10:38:44.897776 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427353 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c4da00 -15480> 2013-04-03 10:38:44.897822 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos -15479> 2013-04-03 10:38:44.897850 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394035 -15478> 2013-04-03 10:38:44.897865 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos latest full 393798 -15477> 2013-04-03 10:38:44.897885 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos applying incremental log 394036 2013-04-03 10:38:43.865726 mon.0 10.252.24.25:6789/0 99 : [INF] pgmap v29033: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 8009B/s wr, 0op/s -15476> 2013-04-03 10:38:44.897930 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 check_subs -15475> 2013-04-03 10:38:44.897962 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 create_pending v 394037 -15474> 2013-04-03 10:38:44.897985 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 _updated_log for mon.0 10.252.24.25:6789/0 -15473> 2013-04-03 10:38:44.897999 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 99) v1 -- ?+0 0x28a5700 con 0x1bac160 -15472> 2013-04-03 10:38:44.898049 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos -15471> 2013-04-03 10:38:44.898065 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 -15470> 2013-04-03 10:38:44.898084 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15469> 2013-04-03 10:38:44.898107 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 99) v1 ==== 0+0+0 (0 0 0) 0x28a5700 con 0x1bac160 -15468> 2013-04-03 10:38:44.898118 7fcdffcdb700 10 handle_log_ack log(last 99) v1 -15467> 2013-04-03 10:38:44.898120 7fcdffcdb700 10 logged 2013-04-03 10:38:43.865726 mon.0 10.252.24.25:6789/0 99 : [INF] pgmap v29033: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 8009B/s wr, 0op/s -15466> 2013-04-03 10:38:44.911174 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113410 ==== paxos(lease_ack lc 427353 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (140852387 0 0) 0x2c4da00 con 0x208a840 -15465> 2013-04-03 10:38:44.911210 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos -15464> 2013-04-03 10:38:44.911227 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 -15463> 2013-04-03 10:38:44.911243 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15462> 2013-04-03 10:38:44.930341 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113411 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 79 v187) v2) to leader v1 ==== 560+0+0 (1856966657 0 0) 0x273a2c0 con 0x208a840 -15461> 2013-04-03 10:38:44.930355 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -15460> 2013-04-03 10:38:44.930364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273a580 from 10.252.24.88:6789/0 -15459> 2013-04-03 10:38:44.930395 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 79 v187) v2 from mds.0 10.252.24.25:6802/3144 -15458> 2013-04-03 10:38:44.930406 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 79 v187) v2 tid 17546) v2 -- ?+0 0x2b26b40 con 0x208a840 -15457> 2013-04-03 10:38:45.383781 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 77 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21735 v187) v2 ==== 289+0+0 (4036472968 0 0) 0x2a83b00 con 0x200d420 -15456> 2013-04-03 10:38:45.383857 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21735 v187) v2 from mds.? 10.252.24.88:6800/17213 -15455> 2013-04-03 10:38:45.383872 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21735 v187) v2 -- ?+0 0x273a2c0 con 0x200d420 -15454> 2013-04-03 10:38:47.416485 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 78 ==== mon_subscribe({mdsmap=188+,monmap=6+}) v2 ==== 42+0+0 (435898880 0 0) 0x2294c40 con 0x200d420 -15453> 2013-04-03 10:38:47.416510 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 -15452> 2013-04-03 10:38:47.416522 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 -15451> 2013-04-03 10:38:47.416526 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/17213 -- mon_subscribe_ack(300s) v1 -- ?+0 0x28a5700 -15450> 2013-04-03 10:38:47.584374 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 78 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140365 v187) v2 ==== 289+0+0 (270333730 0 0) 0x2a528c0 con 0x2118420 -15449> 2013-04-03 10:38:47.584438 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140365 v187) v2 from mds.? 10.252.58.31:6800/10908 -15448> 2013-04-03 10:38:47.584451 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140365 v187) v2 -- ?+0 0x2a83b00 con 0x2118420 -15447> 2013-04-03 10:38:47.897850 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427353 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29fd000 -15446> 2013-04-03 10:38:47.898618 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113412 ==== paxos(lease_ack lc 427353 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (941599202 0 0) 0x29fd000 con 0x208a840 -15445> 2013-04-03 10:38:47.898683 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos -15444> 2013-04-03 10:38:47.898704 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 -15443> 2013-04-03 10:38:47.898721 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15442> 2013-04-03 10:38:48.341866 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 check_down_pgs -15441> 2013-04-03 10:38:48.341899 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 v29033: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 8009B/s wr, 0op/s -15440> 2013-04-03 10:38:48.341927 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -15439> 2013-04-03 10:38:48.341952 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -15438> 2013-04-03 10:38:48.341989 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -15437> 2013-04-03 10:38:48.342021 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos -15436> 2013-04-03 10:38:48.342040 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 -15435> 2013-04-03 10:38:48.342049 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 log -15434> 2013-04-03 10:38:48.342066 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15433> 2013-04-03 10:38:48.342085 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -15432> 2013-04-03 10:38:48.930570 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113413 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 80 v187) v2) to leader v1 ==== 560+0+0 (2660972267 0 0) 0x2996b80 con 0x208a840 -15431> 2013-04-03 10:38:48.930590 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -15430> 2013-04-03 10:38:48.930601 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273b340 from 10.252.24.88:6789/0 -15429> 2013-04-03 10:38:48.930653 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 80 v187) v2 from mds.0 10.252.24.25:6802/3144 -15428> 2013-04-03 10:38:48.930665 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 80 v187) v2 tid 17547) v2 -- ?+0 0x24f8b40 con 0x208a840 -15427> 2013-04-03 10:38:49.383874 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 79 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21736 v187) v2 ==== 289+0+0 (101122530 0 0) 0x295e340 con 0x200d420 -15426> 2013-04-03 10:38:49.383951 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21736 v187) v2 from mds.? 10.252.24.88:6800/17213 -15425> 2013-04-03 10:38:49.383965 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21736 v187) v2 -- ?+0 0x2996b80 con 0x200d420 -15424> 2013-04-03 10:38:50.521234 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2acd780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33065/0 -15423> 2013-04-03 10:38:50.521262 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33065/0) -15422> 2013-04-03 10:38:50.521276 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd780 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -15421> 2013-04-03 10:38:50.521483 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd780 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -15420> 2013-04-03 10:38:50.897924 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427353 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29fd780 -15419> 2013-04-03 10:38:50.898677 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113414 ==== paxos(lease_ack lc 427353 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (694527546 0 0) 0x29fd780 con 0x208a840 -15418> 2013-04-03 10:38:50.898748 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos -15417> 2013-04-03 10:38:50.898769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 -15416> 2013-04-03 10:38:50.898786 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15415> 2013-04-03 10:38:51.584465 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 79 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140366 v187) v2 ==== 289+0+0 (1974417061 0 0) 0x2a51dc0 con 0x2118420 -15414> 2013-04-03 10:38:51.584538 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140366 v187) v2 from mds.? 10.252.58.31:6800/10908 -15413> 2013-04-03 10:38:51.584551 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140366 v187) v2 -- ?+0 0x295e340 con 0x2118420 -15412> 2013-04-03 10:38:51.584563 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113415 ==== forward(pg_stats(1 pgs tid 27 v 605) v1) to leader v1 ==== 753+0+0 (2982828160 0 0) 0x2a528c0 con 0x208a840 -15411> 2013-04-03 10:38:51.584571 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -15410> 2013-04-03 10:38:51.584580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -15409> 2013-04-03 10:38:51.584610 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 preprocess_query pg_stats(1 pgs tid 27 v 605) v1 from osd.1 10.252.24.25:6805/3572 -15408> 2013-04-03 10:38:51.584633 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 prepare_update pg_stats(1 pgs tid 27 v 605) v1 from osd.1 10.252.24.25:6805/3572 -15407> 2013-04-03 10:38:51.584640 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 prepare_pg_stats pg_stats(1 pgs tid 27 v 605) v1 from osd.1 -15406> 2013-04-03 10:38:51.584646 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -15405> 2013-04-03 10:38:51.634832 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 encode_pending v 29034 -15404> 2013-04-03 10:38:51.634867 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427353) queue_proposal bl 650 bytes; ctx = 0x2c3d030 -15403> 2013-04-03 10:38:51.634877 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427353) propose_queued 427354 650 bytes -15402> 2013-04-03 10:38:51.635975 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427353 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd500 -15401> 2013-04-03 10:38:51.673679 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113416 ==== paxos(accept lc 427353 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3241618403 0 0) 0x29fd500 con 0x208a840 -15400> 2013-04-03 10:38:51.673977 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427354 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd780 -15399> 2013-04-03 10:38:51.674004 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427354 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29fd000 -15398> 2013-04-03 10:38:51.674075 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29033 update_from_paxos applying incremental 29034 -15397> 2013-04-03 10:38:51.674092 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 v29034: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2282B/s wr, 0op/s -15396> 2013-04-03 10:38:51.674113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 send_pg_creates to 0 pgs -15395> 2013-04-03 10:38:51.674120 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 update_logger -15394> 2013-04-03 10:38:51.674131 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 create_pending v 29035 -15393> 2013-04-03 10:38:51.674141 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 _updated_stats for osd.1 10.252.24.25:6805/3572 -15392> 2013-04-03 10:38:51.674148 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 27) v1 tid 17548) v2 -- ?+0 0x2b26b40 con 0x208a840 -15391> 2013-04-03 10:38:51.674158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 check_osd_map already seen 605 >= 605 -15390> 2013-04-03 10:38:51.674162 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 update_logger -15389> 2013-04-03 10:38:51.674173 7fcdffcdb700 0 log [INF] : pgmap v29034: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2282B/s wr, 0op/s -15388> 2013-04-03 10:38:51.674174 7fcdffcdb700 10 send_log to self -15387> 2013-04-03 10:38:51.674175 7fcdffcdb700 10 log_queue is 1 last_log 100 sent 99 num 1 unsent 1 sending 1 -15386> 2013-04-03 10:38:51.674177 7fcdffcdb700 10 will send 2013-04-03 10:38:51.674174 mon.0 10.252.24.25:6789/0 100 : [INF] pgmap v29034: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2282B/s wr, 0op/s -15385> 2013-04-03 10:38:51.674183 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74e00 -15384> 2013-04-03 10:38:51.674227 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos -15383> 2013-04-03 10:38:51.674247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 -15382> 2013-04-03 10:38:51.674266 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15381> 2013-04-03 10:38:51.674287 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74e00 con 0x1bac160 -15380> 2013-04-03 10:38:51.674308 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos -15379> 2013-04-03 10:38:51.674324 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 -15378> 2013-04-03 10:38:51.674333 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15377> 2013-04-03 10:38:51.674345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 preprocess_log log(1 entries) v1 from mon.0 -15376> 2013-04-03 10:38:51.674374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15375> 2013-04-03 10:38:51.674388 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 prepare_log log(1 entries) v1 from mon.0 -15374> 2013-04-03 10:38:51.674398 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 logging 2013-04-03 10:38:51.674174 mon.0 10.252.24.25:6789/0 100 : [INF] pgmap v29034: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2282B/s wr, 0op/s -15373> 2013-04-03 10:38:51.699885 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113417 ==== paxos(lease_ack lc 427354 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3955432988 0 0) 0x29fd000 con 0x208a840 -15372> 2013-04-03 10:38:51.699963 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos -15371> 2013-04-03 10:38:51.699984 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 update_from_paxos version 394036 summary v 394036 -15370> 2013-04-03 10:38:51.700001 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15369> 2013-04-03 10:38:52.673879 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394036 encode_pending v394037 -15368> 2013-04-03 10:38:52.673907 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427354) queue_proposal bl 443 bytes; ctx = 0x2c3d150 -15367> 2013-04-03 10:38:52.673915 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427354) propose_queued 427355 443 bytes -15366> 2013-04-03 10:38:52.675382 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427354 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2acda00 -15365> 2013-04-03 10:38:52.700911 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113418 ==== paxos(accept lc 427354 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3320920196 0 0) 0x2acda00 con 0x208a840 -15364> 2013-04-03 10:38:52.701324 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427355 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd000 -15363> 2013-04-03 10:38:52.701348 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427355 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29fd500 -15362> 2013-04-03 10:38:52.701391 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos -15361> 2013-04-03 10:38:52.701428 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394036 -15360> 2013-04-03 10:38:52.701443 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos latest full 393798 -15359> 2013-04-03 10:38:52.701462 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos applying incremental log 394037 2013-04-03 10:38:51.674174 mon.0 10.252.24.25:6789/0 100 : [INF] pgmap v29034: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2282B/s wr, 0op/s -15358> 2013-04-03 10:38:52.701515 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 check_subs -15357> 2013-04-03 10:38:52.701548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 create_pending v 394038 -15356> 2013-04-03 10:38:52.701571 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 _updated_log for mon.0 10.252.24.25:6789/0 -15355> 2013-04-03 10:38:52.701585 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 100) v1 -- ?+0 0x28a5380 con 0x1bac160 -15354> 2013-04-03 10:38:52.701636 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos -15353> 2013-04-03 10:38:52.701655 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 -15352> 2013-04-03 10:38:52.701673 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15351> 2013-04-03 10:38:52.701695 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 100) v1 ==== 0+0+0 (0 0 0) 0x28a5380 con 0x1bac160 -15350> 2013-04-03 10:38:52.701703 7fcdffcdb700 10 handle_log_ack log(last 100) v1 -15349> 2013-04-03 10:38:52.701705 7fcdffcdb700 10 logged 2013-04-03 10:38:51.674174 mon.0 10.252.24.25:6789/0 100 : [INF] pgmap v29034: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2282B/s wr, 0op/s -15348> 2013-04-03 10:38:52.713317 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113419 ==== paxos(lease_ack lc 427355 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3270343031 0 0) 0x29fd500 con 0x208a840 -15347> 2013-04-03 10:38:52.713357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos -15346> 2013-04-03 10:38:52.713375 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 -15345> 2013-04-03 10:38:52.713391 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15344> 2013-04-03 10:38:52.930642 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113420 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 81 v187) v2) to leader v1 ==== 560+0+0 (1985175943 0 0) 0x20e4340 con 0x208a840 -15343> 2013-04-03 10:38:52.930656 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -15342> 2013-04-03 10:38:52.930665 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc5b80 from 10.252.24.88:6789/0 -15341> 2013-04-03 10:38:52.930696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 81 v187) v2 from mds.0 10.252.24.25:6802/3144 -15340> 2013-04-03 10:38:52.930708 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 81 v187) v2 tid 17549) v2 -- ?+0 0x24f8b40 con 0x208a840 -15339> 2013-04-03 10:38:53.332351 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_start_round curr 2 -15338> 2013-04-03 10:38:53.332379 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_start_round new 3 -15337> 2013-04-03 10:38:53.332385 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck -15336> 2013-04-03 10:38:53.332388 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck start timecheck epoch 352 round 3 -15335> 2013-04-03 10:38:53.332400 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck send time_check( ping e 352 r 3 ) v1 to mon.2 10.252.24.88:6789/0 -15334> 2013-04-03 10:38:53.332420 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- time_check( ping e 352 r 3 ) v1 -- ?+0 0x24f8240 -15333> 2013-04-03 10:38:53.332435 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_start_round setting up next event -15332> 2013-04-03 10:38:53.333170 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113421 ==== time_check( pong e 352 r 3 ts 2013-04-03 10:38:53.332248 ) v1 ==== 36+0+0 (3331855384 0 0) 0x24f8240 con 0x208a840 -15331> 2013-04-03 10:38:53.333188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck time_check( pong e 352 r 3 ts 2013-04-03 10:38:53.332248 ) v1 -15330> 2013-04-03 10:38:53.333195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck_leader time_check( pong e 352 r 3 ts 2013-04-03 10:38:53.332248 ) v1 -15329> 2013-04-03 10:38:53.333203 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck_leader from mon.2 10.252.24.88:6789/0 ts 2013-04-03 10:38:53.332248 delta -0.000951052 skew_bound -0.000149052 latency 0.000802 -15328> 2013-04-03 10:38:53.333219 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck_leader got pongs from everybody (2 total) -15327> 2013-04-03 10:38:53.333224 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_finish_round curr 3 -15326> 2013-04-03 10:38:53.333226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_report -15325> 2013-04-03 10:38:53.333231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_report send report to mon.2 10.252.24.88:6789/0 -15324> 2013-04-03 10:38:53.333234 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- time_check( report e 352 r 4 #skews 2 #latencies 2 ) v1 -- ?+0 0x2b26b40 -15323> 2013-04-03 10:38:53.342225 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 check_down_pgs -15322> 2013-04-03 10:38:53.342257 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 v29034: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2282B/s wr, 0op/s -15321> 2013-04-03 10:38:53.342287 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -15320> 2013-04-03 10:38:53.342313 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -15319> 2013-04-03 10:38:53.342692 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -15318> 2013-04-03 10:38:53.342728 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos -15317> 2013-04-03 10:38:53.342747 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 -15316> 2013-04-03 10:38:53.342757 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 log -15315> 2013-04-03 10:38:53.342774 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 update_from_paxos -15314> 2013-04-03 10:38:53.342793 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 auth -15313> 2013-04-03 10:38:53.342826 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 AuthMonitor::tick() updated rotating, now calling propose_pending -15312> 2013-04-03 10:38:53.342891 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2060 encode_pending v 2061 -15311> 2013-04-03 10:38:53.342938 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427355) queue_proposal bl 836 bytes; ctx = 0x1b91610 -15310> 2013-04-03 10:38:53.342950 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427355) propose_queued 427356 836 bytes -15309> 2013-04-03 10:38:53.344488 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427355 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2acdc80 -15308> 2013-04-03 10:38:53.372401 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113422 ==== paxos(accept lc 427355 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3177566905 0 0) 0x2acdc80 con 0x208a840 -15307> 2013-04-03 10:38:53.380808 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427356 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd500 -15306> 2013-04-03 10:38:53.380844 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427356 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2acda00 -15305> 2013-04-03 10:38:53.380884 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -15304> 2013-04-03 10:38:53.380918 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos version 2061 keys ver 2060 latest 2059 -15303> 2013-04-03 10:38:53.380928 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos key server version 2060 -15302> 2013-04-03 10:38:53.380960 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos() last_allocated_id=11896 max_global_id=11896 -15301> 2013-04-03 10:38:53.380979 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 create_pending v 2062 -15300> 2013-04-03 10:38:53.381001 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 AuthMonitor::on_active() -15299> 2013-04-03 10:38:53.381047 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos -15298> 2013-04-03 10:38:53.381064 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 -15297> 2013-04-03 10:38:53.381081 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -15296> 2013-04-03 10:38:53.383939 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 80 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21737 v187) v2 ==== 289+0+0 (627687071 0 0) 0x2a51b00 con 0x200d420 -15295> 2013-04-03 10:38:53.383994 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21737 v187) v2 from mds.? 10.252.24.88:6800/17213 -15294> 2013-04-03 10:38:53.384007 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21737 v187) v2 -- ?+0 0x20e4340 con 0x200d420 -15293> 2013-04-03 10:38:53.406696 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113423 ==== paxos(lease_ack lc 427356 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (4117198255 0 0) 0x2acda00 con 0x208a840 -15292> 2013-04-03 10:38:53.406738 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos -15291> 2013-04-03 10:38:53.406756 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 -15290> 2013-04-03 10:38:53.406773 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -15289> 2013-04-03 10:38:55.481028 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 80 ==== mon_subscribe({mdsmap=188+,monmap=6+}) v2 ==== 42+0+0 (435898880 0 0) 0x28a5e00 con 0x2118420 -15288> 2013-04-03 10:38:55.481050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 -15287> 2013-04-03 10:38:55.481061 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 -15286> 2013-04-03 10:38:55.481064 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.58.31:6800/10908 -- mon_subscribe_ack(300s) v1 -- ?+0 0x28a5380 -15285> 2013-04-03 10:38:55.584596 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 81 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140367 v187) v2 ==== 289+0+0 (1455524312 0 0) 0x273a000 con 0x2118420 -15284> 2013-04-03 10:38:55.584672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140367 v187) v2 from mds.? 10.252.58.31:6800/10908 -15283> 2013-04-03 10:38:55.584687 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140367 v187) v2 -- ?+0 0x2a51b00 con 0x2118420 -15282> 2013-04-03 10:38:56.380919 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427356 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29e0a00 -15281> 2013-04-03 10:38:56.381694 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113424 ==== paxos(lease_ack lc 427356 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2013150657 0 0) 0x29e0a00 con 0x208a840 -15280> 2013-04-03 10:38:56.381761 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos -15279> 2013-04-03 10:38:56.381782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 -15278> 2013-04-03 10:38:56.381800 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -15277> 2013-04-03 10:38:56.930844 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113425 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 82 v187) v2) to leader v1 ==== 560+0+0 (1837695874 0 0) 0x2a528c0 con 0x208a840 -15276> 2013-04-03 10:38:56.930864 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -15275> 2013-04-03 10:38:56.930875 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc58c0 from 10.252.24.88:6789/0 -15274> 2013-04-03 10:38:56.930926 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 82 v187) v2 from mds.0 10.252.24.25:6802/3144 -15273> 2013-04-03 10:38:56.930941 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 82 v187) v2 tid 17550) v2 -- ?+0 0x24f8240 con 0x208a840 -15272> 2013-04-03 10:38:57.384060 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 81 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21738 v187) v2 ==== 289+0+0 (1088061208 0 0) 0x2a52080 con 0x200d420 -15271> 2013-04-03 10:38:57.384129 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21738 v187) v2 from mds.? 10.252.24.88:6800/17213 -15270> 2013-04-03 10:38:57.384143 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21738 v187) v2 -- ?+0 0x2a528c0 con 0x200d420 -15269> 2013-04-03 10:38:58.223533 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113426 ==== forward(pg_stats(1 pgs tid 8 v 605) v1) to leader v1 ==== 757+0+0 (410163318 0 0) 0x273a000 con 0x208a840 -15268> 2013-04-03 10:38:58.223559 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -15267> 2013-04-03 10:38:58.223570 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -15266> 2013-04-03 10:38:58.223623 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 preprocess_query pg_stats(1 pgs tid 8 v 605) v1 from osd.2 10.252.24.30:6801/5656 -15265> 2013-04-03 10:38:58.223658 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 prepare_update pg_stats(1 pgs tid 8 v 605) v1 from osd.2 10.252.24.30:6801/5656 -15264> 2013-04-03 10:38:58.223666 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 prepare_pg_stats pg_stats(1 pgs tid 8 v 605) v1 from osd.2 -15263> 2013-04-03 10:38:58.223673 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -15262> 2013-04-03 10:38:58.273815 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 encode_pending v 29035 -15261> 2013-04-03 10:38:58.273848 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427356) queue_proposal bl 654 bytes; ctx = 0x1b91ee0 -15260> 2013-04-03 10:38:58.273858 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427356) propose_queued 427357 654 bytes -15259> 2013-04-03 10:38:58.275460 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427356 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0c80 -15258> 2013-04-03 10:38:58.319636 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113427 ==== paxos(accept lc 427356 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3426856824 0 0) 0x29e0c80 con 0x208a840 -15257> 2013-04-03 10:38:58.319931 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427357 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0a00 -15256> 2013-04-03 10:38:58.319964 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427357 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2acda00 -15255> 2013-04-03 10:38:58.320033 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29034 update_from_paxos applying incremental 29035 -15254> 2013-04-03 10:38:58.320050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 v29035: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1132B/s wr, 0op/s -15253> 2013-04-03 10:38:58.320071 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 send_pg_creates to 0 pgs -15252> 2013-04-03 10:38:58.320075 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 update_logger -15251> 2013-04-03 10:38:58.320086 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 create_pending v 29036 -15250> 2013-04-03 10:38:58.320097 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 _updated_stats for osd.2 10.252.24.30:6801/5656 -15249> 2013-04-03 10:38:58.320103 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 8) v1 tid 17551) v2 -- ?+0 0x2b26b40 con 0x208a840 -15248> 2013-04-03 10:38:58.320113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 check_osd_map already seen 605 >= 605 -15247> 2013-04-03 10:38:58.320116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 update_logger -15246> 2013-04-03 10:38:58.320126 7fcdffcdb700 0 log [INF] : pgmap v29035: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1132B/s wr, 0op/s -15245> 2013-04-03 10:38:58.320128 7fcdffcdb700 10 send_log to self -15244> 2013-04-03 10:38:58.320129 7fcdffcdb700 10 log_queue is 1 last_log 101 sent 100 num 1 unsent 1 sending 1 -15243> 2013-04-03 10:38:58.320131 7fcdffcdb700 10 will send 2013-04-03 10:38:58.320128 mon.0 10.252.24.25:6789/0 101 : [INF] pgmap v29035: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1132B/s wr, 0op/s -15242> 2013-04-03 10:38:58.320138 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74e00 -15241> 2013-04-03 10:38:58.320186 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos -15240> 2013-04-03 10:38:58.320206 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 -15239> 2013-04-03 10:38:58.320225 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -15238> 2013-04-03 10:38:58.320246 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74e00 con 0x1bac160 -15237> 2013-04-03 10:38:58.320268 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos -15236> 2013-04-03 10:38:58.320283 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 -15235> 2013-04-03 10:38:58.320292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15234> 2013-04-03 10:38:58.320304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 preprocess_log log(1 entries) v1 from mon.0 -15233> 2013-04-03 10:38:58.320328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15232> 2013-04-03 10:38:58.320341 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 prepare_log log(1 entries) v1 from mon.0 -15231> 2013-04-03 10:38:58.320354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 logging 2013-04-03 10:38:58.320128 mon.0 10.252.24.25:6789/0 101 : [INF] pgmap v29035: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1132B/s wr, 0op/s -15230> 2013-04-03 10:38:58.332005 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113428 ==== paxos(lease_ack lc 427357 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2551372230 0 0) 0x2acda00 con 0x208a840 -15229> 2013-04-03 10:38:58.332040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos -15228> 2013-04-03 10:38:58.332056 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 -15227> 2013-04-03 10:38:58.332073 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -15226> 2013-04-03 10:38:58.344630 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 check_down_pgs -15225> 2013-04-03 10:38:58.344666 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29035 v29035: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1132B/s wr, 0op/s -15224> 2013-04-03 10:38:58.344692 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -15223> 2013-04-03 10:38:58.344713 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -15222> 2013-04-03 10:38:58.344751 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -15221> 2013-04-03 10:38:58.344781 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos -15220> 2013-04-03 10:38:58.344799 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 update_from_paxos version 394037 summary v 394037 -15219> 2013-04-03 10:38:58.344808 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394037 log -15218> 2013-04-03 10:38:58.344824 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -15217> 2013-04-03 10:38:58.344842 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -15216> 2013-04-03 10:38:58.429147 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) -15215> 2013-04-03 10:38:58.429172 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader -15214> 2013-04-03 10:38:59.319842 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394037 encode_pending v394038 -15213> 2013-04-03 10:38:59.319871 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427357) queue_proposal bl 443 bytes; ctx = 0x2c3d400 -15212> 2013-04-03 10:38:59.319879 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427357) propose_queued 427358 443 bytes -15211> 2013-04-03 10:38:59.321150 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427357 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0000 -15210> 2013-04-03 10:38:59.434157 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113429 ==== paxos(accept lc 427357 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (68371590 0 0) 0x29e0000 con 0x208a840 -15209> 2013-04-03 10:38:59.434448 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427358 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2acda00 -15208> 2013-04-03 10:38:59.434484 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427358 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29e0c80 -15207> 2013-04-03 10:38:59.434523 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos -15206> 2013-04-03 10:38:59.434550 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394037 -15205> 2013-04-03 10:38:59.434564 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos latest full 393798 -15204> 2013-04-03 10:38:59.434582 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394038 update_from_paxos applying incremental log 394038 2013-04-03 10:38:58.320128 mon.0 10.252.24.25:6789/0 101 : [INF] pgmap v29035: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1132B/s wr, 0op/s -15203> 2013-04-03 10:38:59.434628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 check_subs -15202> 2013-04-03 10:38:59.434659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 create_pending v 394039 -15201> 2013-04-03 10:38:59.434681 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394038 _updated_log for mon.0 10.252.24.25:6789/0 -15200> 2013-04-03 10:38:59.434693 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 101) v1 -- ?+0 0x28a5540 con 0x1bac160 -15199> 2013-04-03 10:38:59.434744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos -15198> 2013-04-03 10:38:59.434760 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 -15197> 2013-04-03 10:38:59.434777 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -15196> 2013-04-03 10:38:59.434804 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 101) v1 ==== 0+0+0 (0 0 0) 0x28a5540 con 0x1bac160 -15195> 2013-04-03 10:38:59.434812 7fcdffcdb700 10 handle_log_ack log(last 101) v1 -15194> 2013-04-03 10:38:59.434814 7fcdffcdb700 10 logged 2013-04-03 10:38:58.320128 mon.0 10.252.24.25:6789/0 101 : [INF] pgmap v29035: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1132B/s wr, 0op/s -15193> 2013-04-03 10:38:59.460331 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113430 ==== paxos(lease_ack lc 427358 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2317155644 0 0) 0x29e0c80 con 0x208a840 -15192> 2013-04-03 10:38:59.460371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos -15191> 2013-04-03 10:38:59.460802 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 -15190> 2013-04-03 10:38:59.460826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -15189> 2013-04-03 10:38:59.584737 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 82 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140368 v187) v2 ==== 289+0+0 (2640454272 0 0) 0x2ce2b80 con 0x2118420 -15188> 2013-04-03 10:38:59.584788 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140368 v187) v2 from mds.? 10.252.58.31:6800/10908 -15187> 2013-04-03 10:38:59.584800 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140368 v187) v2 -- ?+0 0x273a000 con 0x2118420 -15186> 2013-04-03 10:39:00.537189 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2acd500 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33066/0 -15185> 2013-04-03 10:39:00.537219 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd500 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33066/0) -15184> 2013-04-03 10:39:00.537235 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd500 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -15183> 2013-04-03 10:39:00.537440 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd500 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -15182> 2013-04-03 10:39:00.930951 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113431 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 83 v187) v2) to leader v1 ==== 560+0+0 (953071736 0 0) 0x2cc5600 con 0x208a840 -15181> 2013-04-03 10:39:00.930973 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -15180> 2013-04-03 10:39:00.930984 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc5340 from 10.252.24.88:6789/0 -15179> 2013-04-03 10:39:00.931036 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 83 v187) v2 from mds.0 10.252.24.25:6802/3144 -15178> 2013-04-03 10:39:00.931047 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 83 v187) v2 tid 17552) v2 -- ?+0 0x24f8240 con 0x208a840 -15177> 2013-04-03 10:39:01.384193 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 82 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21739 v187) v2 ==== 289+0+0 (1672790117 0 0) 0x29c4000 con 0x200d420 -15176> 2013-04-03 10:39:01.384270 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21739 v187) v2 from mds.? 10.252.24.88:6800/17213 -15175> 2013-04-03 10:39:01.384283 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21739 v187) v2 -- ?+0 0x2cc5600 con 0x200d420 -15174> 2013-04-03 10:39:02.434560 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427358 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29b4280 -15173> 2013-04-03 10:39:02.435369 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113432 ==== paxos(lease_ack lc 427358 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3936862586 0 0) 0x29b4280 con 0x208a840 -15172> 2013-04-03 10:39:02.435445 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos -15171> 2013-04-03 10:39:02.435465 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 -15170> 2013-04-03 10:39:02.435482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -15169> 2013-04-03 10:39:03.344982 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29035 v29035: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1132B/s wr, 0op/s -15168> 2013-04-03 10:39:03.345031 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -15167> 2013-04-03 10:39:03.345056 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -15166> 2013-04-03 10:39:03.345100 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -15165> 2013-04-03 10:39:03.345134 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos -15164> 2013-04-03 10:39:03.345153 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 -15163> 2013-04-03 10:39:03.345161 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394038 log -15162> 2013-04-03 10:39:03.345177 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -15161> 2013-04-03 10:39:03.345196 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -15160> 2013-04-03 10:39:03.584899 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 83 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140369 v187) v2 ==== 289+0+0 (3188497917 0 0) 0x2ce28c0 con 0x2118420 -15159> 2013-04-03 10:39:03.584974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140369 v187) v2 from mds.? 10.252.58.31:6800/10908 -15158> 2013-04-03 10:39:03.584986 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140369 v187) v2 -- ?+0 0x29c4000 con 0x2118420 -15157> 2013-04-03 10:39:04.931090 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113433 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 84 v187) v2) to leader v1 ==== 560+0+0 (2879808044 0 0) 0x2ce2b80 con 0x208a840 -15156> 2013-04-03 10:39:04.931111 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -15155> 2013-04-03 10:39:04.931121 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc5080 from 10.252.24.88:6789/0 -15154> 2013-04-03 10:39:04.931173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 84 v187) v2 from mds.0 10.252.24.25:6802/3144 -15153> 2013-04-03 10:39:04.931184 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 84 v187) v2 tid 17553) v2 -- ?+0 0x2b26b40 con 0x208a840 -15152> 2013-04-03 10:39:05.384283 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 83 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21740 v187) v2 ==== 289+0+0 (2344483862 0 0) 0x29c5340 con 0x200d420 -15151> 2013-04-03 10:39:05.384358 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21740 v187) v2 from mds.? 10.252.24.88:6800/17213 -15150> 2013-04-03 10:39:05.384371 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21740 v187) v2 -- ?+0 0x2ce2b80 con 0x200d420 -15149> 2013-04-03 10:39:05.434623 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427358 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29b4000 -15148> 2013-04-03 10:39:05.435398 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113434 ==== paxos(lease_ack lc 427358 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3553858196 0 0) 0x29b4000 con 0x208a840 -15147> 2013-04-03 10:39:05.435448 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos -15146> 2013-04-03 10:39:05.435468 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 -15145> 2013-04-03 10:39:05.435486 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -15144> 2013-04-03 10:39:06.586252 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113435 ==== forward(pg_stats(1 pgs tid 28 v 605) v1) to leader v1 ==== 753+0+0 (1058209480 0 0) 0x2ce28c0 con 0x208a840 -15143> 2013-04-03 10:39:06.586274 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -15142> 2013-04-03 10:39:06.586284 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 -15141> 2013-04-03 10:39:06.586336 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29035 preprocess_query pg_stats(1 pgs tid 28 v 605) v1 from osd.1 10.252.24.25:6805/3572 -15140> 2013-04-03 10:39:06.586360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29035 prepare_update pg_stats(1 pgs tid 28 v 605) v1 from osd.1 10.252.24.25:6805/3572 -15139> 2013-04-03 10:39:06.586365 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29035 prepare_pg_stats pg_stats(1 pgs tid 28 v 605) v1 from osd.1 -15138> 2013-04-03 10:39:06.586371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29035 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -15137> 2013-04-03 10:39:06.636556 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29035 encode_pending v 29036 -15136> 2013-04-03 10:39:06.636590 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427358) queue_proposal bl 650 bytes; ctx = 0x2c3d3f0 -15135> 2013-04-03 10:39:06.636599 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427358) propose_queued 427359 650 bytes -15134> 2013-04-03 10:39:06.637823 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427358 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc280 -15133> 2013-04-03 10:39:06.666008 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113436 ==== paxos(accept lc 427358 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (18183930 0 0) 0x26dc280 con 0x208a840 -15132> 2013-04-03 10:39:06.672707 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427359 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4000 -15131> 2013-04-03 10:39:06.672737 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427359 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29b4280 -15130> 2013-04-03 10:39:06.672806 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29035 update_from_paxos applying incremental 29036 -15129> 2013-04-03 10:39:06.672823 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 v29036: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -15128> 2013-04-03 10:39:06.672842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 send_pg_creates to 0 pgs -15127> 2013-04-03 10:39:06.672847 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 update_logger -15126> 2013-04-03 10:39:06.672857 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 create_pending v 29037 -15125> 2013-04-03 10:39:06.672866 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29036 _updated_stats for osd.1 10.252.24.25:6805/3572 -15124> 2013-04-03 10:39:06.672874 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 28) v1 tid 17554) v2 -- ?+0 0x24f8240 con 0x208a840 -15123> 2013-04-03 10:39:06.672884 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 check_osd_map already seen 605 >= 605 -15122> 2013-04-03 10:39:06.672887 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 update_logger -15121> 2013-04-03 10:39:06.672897 7fcdffcdb700 0 log [INF] : pgmap v29036: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -15120> 2013-04-03 10:39:06.672899 7fcdffcdb700 10 send_log to self -15119> 2013-04-03 10:39:06.672900 7fcdffcdb700 10 log_queue is 1 last_log 102 sent 101 num 1 unsent 1 sending 1 -15118> 2013-04-03 10:39:06.672901 7fcdffcdb700 10 will send 2013-04-03 10:39:06.672898 mon.0 10.252.24.25:6789/0 102 : [INF] pgmap v29036: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -15117> 2013-04-03 10:39:06.672908 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74e00 -15116> 2013-04-03 10:39:06.672956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos -15115> 2013-04-03 10:39:06.672975 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 -15114> 2013-04-03 10:39:06.672993 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -15113> 2013-04-03 10:39:06.673014 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74e00 con 0x1bac160 -15112> 2013-04-03 10:39:06.673036 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos -15111> 2013-04-03 10:39:06.673050 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 -15110> 2013-04-03 10:39:06.673058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15109> 2013-04-03 10:39:06.673068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 preprocess_log log(1 entries) v1 from mon.0 -15108> 2013-04-03 10:39:06.673092 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15107> 2013-04-03 10:39:06.673105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 prepare_log log(1 entries) v1 from mon.0 -15106> 2013-04-03 10:39:06.673114 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 logging 2013-04-03 10:39:06.672898 mon.0 10.252.24.25:6789/0 102 : [INF] pgmap v29036: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -15105> 2013-04-03 10:39:06.687857 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113437 ==== paxos(lease_ack lc 427359 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3775793871 0 0) 0x29b4280 con 0x208a840 -15104> 2013-04-03 10:39:06.687896 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos -15103> 2013-04-03 10:39:06.687913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394038 update_from_paxos version 394038 summary v 394038 -15102> 2013-04-03 10:39:06.687929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -15101> 2013-04-03 10:39:06.751628 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113438 ==== forward(pg_stats(1 pgs tid 30445 v 605) v1) to leader v1 ==== 757+0+0 (3105936677 0 0) 0x2cc4dc0 con 0x208a840 -15100> 2013-04-03 10:39:06.751641 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -15099> 2013-04-03 10:39:06.751649 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -15098> 2013-04-03 10:39:06.751678 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 preprocess_query pg_stats(1 pgs tid 30445 v 605) v1 from osd.3 10.252.24.88:6801/17538 -15097> 2013-04-03 10:39:06.751698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 prepare_update pg_stats(1 pgs tid 30445 v 605) v1 from osd.3 10.252.24.88:6801/17538 -15096> 2013-04-03 10:39:06.751704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 prepare_pg_stats pg_stats(1 pgs tid 30445 v 605) v1 from osd.3 -15095> 2013-04-03 10:39:06.751708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29036 got osd.3 osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -15094> 2013-04-03 10:39:07.585009 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 84 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140370 v187) v2 ==== 289+0+0 (3686751354 0 0) 0x2ce2600 con 0x2118420 -15093> 2013-04-03 10:39:07.585084 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140370 v187) v2 from mds.? 10.252.58.31:6800/10908 -15092> 2013-04-03 10:39:07.585097 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140370 v187) v2 -- ?+0 0x2cc4dc0 con 0x2118420 -15091> 2013-04-03 10:39:07.666188 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29036 encode_pending v 29037 -15090> 2013-04-03 10:39:07.666218 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427359) queue_proposal bl 654 bytes; ctx = 0x2c3d130 -15089> 2013-04-03 10:39:07.666227 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427359) propose_queued 427360 654 bytes -15088> 2013-04-03 10:39:07.668036 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427359 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4c80 -15087> 2013-04-03 10:39:07.668132 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394038 encode_pending v394039 -15086> 2013-04-03 10:39:07.668158 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427326..427359) queue_proposal bl 443 bytes; ctx = 0x2c3d020 -15085> 2013-04-03 10:39:07.668164 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427326..427359) propose_new_value not active; proposal queued -15084> 2013-04-03 10:39:07.697378 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113439 ==== paxos(accept lc 427359 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2476431325 0 0) 0x20f4c80 con 0x208a840 -15083> 2013-04-03 10:39:07.697708 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427360 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4280 -15082> 2013-04-03 10:39:07.697734 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427360 fc 427326 pn 0 opn 0) v3 -- ?+0 0x26dc280 -15081> 2013-04-03 10:39:07.697809 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29036 update_from_paxos applying incremental 29037 -15080> 2013-04-03 10:39:07.697837 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 v29037: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -15079> 2013-04-03 10:39:07.697858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 send_pg_creates to 0 pgs -15078> 2013-04-03 10:39:07.697861 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 update_logger -15077> 2013-04-03 10:39:07.697872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 create_pending v 29038 -15076> 2013-04-03 10:39:07.697881 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29037 _updated_stats for osd.3 10.252.24.88:6801/17538 -15075> 2013-04-03 10:39:07.697887 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30445) v1 tid 17555) v2 -- ?+0 0x2b26b40 con 0x208a840 -15074> 2013-04-03 10:39:07.697897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 check_osd_map already seen 605 >= 605 -15073> 2013-04-03 10:39:07.697900 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 update_logger -15072> 2013-04-03 10:39:07.697911 7fcdffcdb700 0 log [INF] : pgmap v29037: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -15071> 2013-04-03 10:39:07.697912 7fcdffcdb700 10 send_log to self -15070> 2013-04-03 10:39:07.697913 7fcdffcdb700 10 log_queue is 2 last_log 103 sent 102 num 2 unsent 1 sending 1 -15069> 2013-04-03 10:39:07.697914 7fcdffcdb700 10 will send 2013-04-03 10:39:06.672898 mon.0 10.252.24.25:6789/0 102 : [INF] pgmap v29036: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -15068> 2013-04-03 10:39:07.697921 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74600 -15067> 2013-04-03 10:39:07.697946 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427360) propose_queued 427361 443 bytes -15066> 2013-04-03 10:39:07.698155 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427360 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0c80 -15065> 2013-04-03 10:39:07.698188 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74600 con 0x1bac160 -15064> 2013-04-03 10:39:07.725184 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113440 ==== paxos(lease_ack lc 427360 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1811812901 0 0) 0x29e0c80 con 0x208a840 -15063> 2013-04-03 10:39:07.737463 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113441 ==== paxos(accept lc 427360 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (93952023 0 0) 0x26dc280 con 0x208a840 -15062> 2013-04-03 10:39:07.737751 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427361 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0c80 -15061> 2013-04-03 10:39:07.737785 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427361 fc 427326 pn 0 opn 0) v3 -- ?+0 0x20f4c80 -15060> 2013-04-03 10:39:07.737829 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos -15059> 2013-04-03 10:39:07.737852 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394038 -15058> 2013-04-03 10:39:07.737866 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos latest full 393798 -15057> 2013-04-03 10:39:07.737884 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394039 update_from_paxos applying incremental log 394039 2013-04-03 10:39:06.672898 mon.0 10.252.24.25:6789/0 102 : [INF] pgmap v29036: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -15056> 2013-04-03 10:39:07.737930 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 check_subs -15055> 2013-04-03 10:39:07.737961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 create_pending v 394040 -15054> 2013-04-03 10:39:07.738257 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394039 _updated_log for mon.0 10.252.24.25:6789/0 -15053> 2013-04-03 10:39:07.738270 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 102) v1 -- ?+0 0x28a5c40 con 0x1bac160 -15052> 2013-04-03 10:39:07.738291 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos -15051> 2013-04-03 10:39:07.738305 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 -15050> 2013-04-03 10:39:07.738315 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -15049> 2013-04-03 10:39:07.738327 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 preprocess_log log(1 entries) v1 from mon.0 -15048> 2013-04-03 10:39:07.738345 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 nothing new -15047> 2013-04-03 10:39:07.738391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos -15046> 2013-04-03 10:39:07.738406 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 -15045> 2013-04-03 10:39:07.738424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -15044> 2013-04-03 10:39:07.738448 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 102) v1 ==== 0+0+0 (0 0 0) 0x28a5c40 con 0x1bac160 -15043> 2013-04-03 10:39:07.738456 7fcdffcdb700 10 handle_log_ack log(last 102) v1 -15042> 2013-04-03 10:39:07.738457 7fcdffcdb700 10 logged 2013-04-03 10:39:06.672898 mon.0 10.252.24.25:6789/0 102 : [INF] pgmap v29036: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -15041> 2013-04-03 10:39:07.751170 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113442 ==== paxos(lease_ack lc 427361 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2512875845 0 0) 0x20f4c80 con 0x208a840 -15040> 2013-04-03 10:39:07.751211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos -15039> 2013-04-03 10:39:07.751228 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 -15038> 2013-04-03 10:39:07.751244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -15037> 2013-04-03 10:39:08.345350 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29037 check_down_pgs -15036> 2013-04-03 10:39:08.345379 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29037 v29037: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -15035> 2013-04-03 10:39:08.345410 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -15034> 2013-04-03 10:39:08.345434 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -15033> 2013-04-03 10:39:08.345470 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -15032> 2013-04-03 10:39:08.345501 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos -15031> 2013-04-03 10:39:08.345999 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 -15030> 2013-04-03 10:39:08.346017 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394039 log -15029> 2013-04-03 10:39:08.346035 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -15028> 2013-04-03 10:39:08.346054 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -15027> 2013-04-03 10:39:08.931227 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113443 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 85 v187) v2) to leader v1 ==== 560+0+0 (2538794518 0 0) 0x2cc4b00 con 0x208a840 -15026> 2013-04-03 10:39:08.931249 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -15025> 2013-04-03 10:39:08.931261 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4840 from 10.252.24.88:6789/0 -15024> 2013-04-03 10:39:08.931314 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 85 v187) v2 from mds.0 10.252.24.25:6802/3144 -15023> 2013-04-03 10:39:08.931325 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 85 v187) v2 tid 17556) v2 -- ?+0 0x24f8240 con 0x208a840 -15022> 2013-04-03 10:39:09.384386 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 84 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21741 v187) v2 ==== 289+0+0 (2832417643 0 0) 0x29c4580 con 0x200d420 -15021> 2013-04-03 10:39:09.384461 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21741 v187) v2 from mds.? 10.252.24.88:6800/17213 -15020> 2013-04-03 10:39:09.384475 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21741 v187) v2 -- ?+0 0x2cc4b00 con 0x200d420 -15019> 2013-04-03 10:39:10.737865 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427361 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2ab4500 -15018> 2013-04-03 10:39:10.738672 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113444 ==== paxos(lease_ack lc 427361 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2687018926 0 0) 0x2ab4500 con 0x208a840 -15017> 2013-04-03 10:39:10.738740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos -15016> 2013-04-03 10:39:10.738759 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 -15015> 2013-04-03 10:39:10.738776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -15014> 2013-04-03 10:39:11.585133 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 85 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140371 v187) v2 ==== 289+0+0 (4174504711 0 0) 0x2ce2340 con 0x2118420 -15013> 2013-04-03 10:39:11.585206 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140371 v187) v2 from mds.? 10.252.58.31:6800/10908 -15012> 2013-04-03 10:39:11.585218 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140371 v187) v2 -- ?+0 0x29c4580 con 0x2118420 -15011> 2013-04-03 10:39:12.931394 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113445 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 86 v187) v2) to leader v1 ==== 560+0+0 (346037879 0 0) 0x2ce2600 con 0x208a840 -15010> 2013-04-03 10:39:12.931415 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -15009> 2013-04-03 10:39:12.931425 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4580 from 10.252.24.88:6789/0 -15008> 2013-04-03 10:39:12.931479 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 86 v187) v2 from mds.0 10.252.24.25:6802/3144 -15007> 2013-04-03 10:39:12.931489 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 86 v187) v2 tid 17557) v2 -- ?+0 0x2b26b40 con 0x208a840 -15006> 2013-04-03 10:39:13.225822 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113446 ==== forward(pg_stats(1 pgs tid 9 v 605) v1) to leader v1 ==== 757+0+0 (38632899 0 0) 0x2ce2340 con 0x208a840 -15005> 2013-04-03 10:39:13.225848 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -15004> 2013-04-03 10:39:13.225858 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 -15003> 2013-04-03 10:39:13.225910 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 preprocess_query pg_stats(1 pgs tid 9 v 605) v1 from osd.2 10.252.24.30:6801/5656 -15002> 2013-04-03 10:39:13.225934 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 prepare_update pg_stats(1 pgs tid 9 v 605) v1 from osd.2 10.252.24.30:6801/5656 -15001> 2013-04-03 10:39:13.225940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 prepare_pg_stats pg_stats(1 pgs tid 9 v 605) v1 from osd.2 -15000> 2013-04-03 10:39:13.225946 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29037 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -14999> 2013-04-03 10:39:13.276136 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29037 encode_pending v 29038 -14998> 2013-04-03 10:39:13.276169 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427361) queue_proposal bl 654 bytes; ctx = 0x2c3d320 -14997> 2013-04-03 10:39:13.276178 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427361) propose_queued 427362 654 bytes -14996> 2013-04-03 10:39:13.277129 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427361 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4780 -14995> 2013-04-03 10:39:13.307162 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113447 ==== paxos(accept lc 427361 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (796413733 0 0) 0x2ab4780 con 0x208a840 -14994> 2013-04-03 10:39:13.308005 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427362 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4500 -14993> 2013-04-03 10:39:13.308023 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427362 fc 427326 pn 0 opn 0) v3 -- ?+0 0x20f4c80 -14992> 2013-04-03 10:39:13.308089 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29037 update_from_paxos applying incremental 29038 -14991> 2013-04-03 10:39:13.308105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 v29038: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2621B/s wr, 0op/s -14990> 2013-04-03 10:39:13.308125 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 send_pg_creates to 0 pgs -14989> 2013-04-03 10:39:13.308127 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 update_logger -14988> 2013-04-03 10:39:13.308139 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 create_pending v 29039 -14987> 2013-04-03 10:39:13.308152 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29038 _updated_stats for osd.2 10.252.24.30:6801/5656 -14986> 2013-04-03 10:39:13.308157 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 9) v1 tid 17558) v2 -- ?+0 0x24f8240 con 0x208a840 -14985> 2013-04-03 10:39:13.308166 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 check_osd_map already seen 605 >= 605 -14984> 2013-04-03 10:39:13.308169 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 update_logger -14983> 2013-04-03 10:39:13.308180 7fcdffcdb700 0 log [INF] : pgmap v29038: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2621B/s wr, 0op/s -14982> 2013-04-03 10:39:13.308181 7fcdffcdb700 10 send_log to self -14981> 2013-04-03 10:39:13.308182 7fcdffcdb700 10 log_queue is 2 last_log 104 sent 102 num 2 unsent 2 sending 2 -14980> 2013-04-03 10:39:13.308184 7fcdffcdb700 10 will send 2013-04-03 10:39:07.697912 mon.0 10.252.24.25:6789/0 103 : [INF] pgmap v29037: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -14979> 2013-04-03 10:39:13.308189 7fcdffcdb700 10 will send 2013-04-03 10:39:13.308181 mon.0 10.252.24.25:6789/0 104 : [INF] pgmap v29038: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2621B/s wr, 0op/s -14978> 2013-04-03 10:39:13.308195 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b74600 -14977> 2013-04-03 10:39:13.308241 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos -14976> 2013-04-03 10:39:13.308259 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 -14975> 2013-04-03 10:39:13.308278 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14974> 2013-04-03 10:39:13.308298 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74600 con 0x1bac160 -14973> 2013-04-03 10:39:13.308320 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos -14972> 2013-04-03 10:39:13.308334 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 -14971> 2013-04-03 10:39:13.308343 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -14970> 2013-04-03 10:39:13.308354 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 preprocess_log log(2 entries) v1 from mon.0 -14969> 2013-04-03 10:39:13.308386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -14968> 2013-04-03 10:39:13.308399 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 prepare_log log(2 entries) v1 from mon.0 -14967> 2013-04-03 10:39:13.308408 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 logging 2013-04-03 10:39:07.697912 mon.0 10.252.24.25:6789/0 103 : [INF] pgmap v29037: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -14966> 2013-04-03 10:39:13.308429 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 logging 2013-04-03 10:39:13.308181 mon.0 10.252.24.25:6789/0 104 : [INF] pgmap v29038: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2621B/s wr, 0op/s -14965> 2013-04-03 10:39:13.333221 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113448 ==== paxos(lease_ack lc 427362 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2548709282 0 0) 0x20f4c80 con 0x208a840 -14964> 2013-04-03 10:39:13.333263 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos -14963> 2013-04-03 10:39:13.333279 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 -14962> 2013-04-03 10:39:13.333294 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14961> 2013-04-03 10:39:13.346162 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29038 check_down_pgs -14960> 2013-04-03 10:39:13.346192 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29038 v29038: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2621B/s wr, 0op/s -14959> 2013-04-03 10:39:13.346218 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -14958> 2013-04-03 10:39:13.346238 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -14957> 2013-04-03 10:39:13.346274 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -14956> 2013-04-03 10:39:13.346304 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos -14955> 2013-04-03 10:39:13.346322 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394039 update_from_paxos version 394039 summary v 394039 -14954> 2013-04-03 10:39:13.346330 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394039 log -14953> 2013-04-03 10:39:13.346346 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14952> 2013-04-03 10:39:13.346362 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -14951> 2013-04-03 10:39:13.384470 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 85 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21742 v187) v2 ==== 289+0+0 (3445654252 0 0) 0x2cc5340 con 0x200d420 -14950> 2013-04-03 10:39:13.384518 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21742 v187) v2 from mds.? 10.252.24.88:6800/17213 -14949> 2013-04-03 10:39:13.384527 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21742 v187) v2 -- ?+0 0x2ce2340 con 0x200d420 -14948> 2013-04-03 10:39:14.307367 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394039 encode_pending v394040 -14947> 2013-04-03 10:39:14.307397 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427362) queue_proposal bl 725 bytes; ctx = 0x2c3d120 -14946> 2013-04-03 10:39:14.307405 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427362) propose_queued 427363 725 bytes -14945> 2013-04-03 10:39:14.308487 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427362 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4280 -14944> 2013-04-03 10:39:14.333306 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113449 ==== paxos(accept lc 427362 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3354395906 0 0) 0x2ab4280 con 0x208a840 -14943> 2013-04-03 10:39:14.333633 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427363 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4c80 -14942> 2013-04-03 10:39:14.333666 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427363 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2ab4780 -14941> 2013-04-03 10:39:14.333702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos -14940> 2013-04-03 10:39:14.333728 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394039 -14939> 2013-04-03 10:39:14.333744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos latest full 393798 -14938> 2013-04-03 10:39:14.333763 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394040 update_from_paxos applying incremental log 394040 2013-04-03 10:39:07.697912 mon.0 10.252.24.25:6789/0 103 : [INF] pgmap v29037: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -14937> 2013-04-03 10:39:14.333792 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394040 update_from_paxos applying incremental log 394040 2013-04-03 10:39:13.308181 mon.0 10.252.24.25:6789/0 104 : [INF] pgmap v29038: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2621B/s wr, 0op/s -14936> 2013-04-03 10:39:14.334037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 check_subs -14935> 2013-04-03 10:39:14.334074 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 create_pending v 394041 -14934> 2013-04-03 10:39:14.334097 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394040 _updated_log for mon.0 10.252.24.25:6789/0 -14933> 2013-04-03 10:39:14.334111 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 104) v1 -- ?+0 0x2224700 con 0x1bac160 -14932> 2013-04-03 10:39:14.334163 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos -14931> 2013-04-03 10:39:14.334178 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 -14930> 2013-04-03 10:39:14.334196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14929> 2013-04-03 10:39:14.334218 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 104) v1 ==== 0+0+0 (0 0 0) 0x2224700 con 0x1bac160 -14928> 2013-04-03 10:39:14.334227 7fcdffcdb700 10 handle_log_ack log(last 104) v1 -14927> 2013-04-03 10:39:14.334229 7fcdffcdb700 10 logged 2013-04-03 10:39:07.697912 mon.0 10.252.24.25:6789/0 103 : [INF] pgmap v29037: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -14926> 2013-04-03 10:39:14.334236 7fcdffcdb700 10 logged 2013-04-03 10:39:13.308181 mon.0 10.252.24.25:6789/0 104 : [INF] pgmap v29038: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2621B/s wr, 0op/s -14925> 2013-04-03 10:39:14.346964 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113450 ==== paxos(lease_ack lc 427363 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3781845600 0 0) 0x2ab4780 con 0x208a840 -14924> 2013-04-03 10:39:14.347002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos -14923> 2013-04-03 10:39:14.347018 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 -14922> 2013-04-03 10:39:14.347034 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14921> 2013-04-03 10:39:15.585233 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 86 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140372 v187) v2 ==== 289+0+0 (282648436 0 0) 0x2ce2080 con 0x2118420 -14920> 2013-04-03 10:39:15.585321 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140372 v187) v2 from mds.? 10.252.58.31:6800/10908 -14919> 2013-04-03 10:39:15.585335 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140372 v187) v2 -- ?+0 0x2cc5340 con 0x2118420 -14918> 2013-04-03 10:39:16.931512 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113451 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 87 v187) v2) to leader v1 ==== 560+0+0 (3386186530 0 0) 0x2cc42c0 con 0x208a840 -14917> 2013-04-03 10:39:16.931536 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -14916> 2013-04-03 10:39:16.931546 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4000 from 10.252.24.88:6789/0 -14915> 2013-04-03 10:39:16.931599 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 87 v187) v2 from mds.0 10.252.24.25:6802/3144 -14914> 2013-04-03 10:39:16.931611 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 87 v187) v2 tid 17559) v2 -- ?+0 0x24f8900 con 0x208a840 -14913> 2013-04-03 10:39:17.333745 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427363 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2ab4000 -14912> 2013-04-03 10:39:17.334553 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113452 ==== paxos(lease_ack lc 427363 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2513557861 0 0) 0x2ab4000 con 0x208a840 -14911> 2013-04-03 10:39:17.334629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos -14910> 2013-04-03 10:39:17.334649 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 -14909> 2013-04-03 10:39:17.334666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14908> 2013-04-03 10:39:17.384643 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 86 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21743 v187) v2 ==== 289+0+0 (3993910673 0 0) 0x2cc58c0 con 0x200d420 -14907> 2013-04-03 10:39:17.384690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21743 v187) v2 from mds.? 10.252.24.88:6800/17213 -14906> 2013-04-03 10:39:17.384702 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21743 v187) v2 -- ?+0 0x2cc42c0 con 0x200d420 -14905> 2013-04-03 10:39:18.346511 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29038 v29038: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2621B/s wr, 0op/s -14904> 2013-04-03 10:39:18.346551 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -14903> 2013-04-03 10:39:18.346574 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -14902> 2013-04-03 10:39:18.346616 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -14901> 2013-04-03 10:39:18.346647 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos -14900> 2013-04-03 10:39:18.346666 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 -14899> 2013-04-03 10:39:18.346675 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394040 log -14898> 2013-04-03 10:39:18.346695 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14897> 2013-04-03 10:39:18.346714 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -14896> 2013-04-03 10:39:19.585379 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 87 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140373 v187) v2 ==== 289+0+0 (867590153 0 0) 0x2ce1dc0 con 0x2118420 -14895> 2013-04-03 10:39:19.585450 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140373 v187) v2 from mds.? 10.252.58.31:6800/10908 -14894> 2013-04-03 10:39:19.585463 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140373 v187) v2 -- ?+0 0x2cc58c0 con 0x2118420 -14893> 2013-04-03 10:39:20.333819 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427363 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2787500 -14892> 2013-04-03 10:39:20.334636 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113453 ==== paxos(lease_ack lc 427363 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3802016117 0 0) 0x2787500 con 0x208a840 -14891> 2013-04-03 10:39:20.334710 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos -14890> 2013-04-03 10:39:20.334730 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 -14889> 2013-04-03 10:39:20.334747 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14888> 2013-04-03 10:39:20.931576 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113454 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 88 v187) v2) to leader v1 ==== 560+0+0 (3177640243 0 0) 0x2ce2080 con 0x208a840 -14887> 2013-04-03 10:39:20.931595 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -14886> 2013-04-03 10:39:20.931603 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52600 from 10.252.24.88:6789/0 -14885> 2013-04-03 10:39:20.931636 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 88 v187) v2 from mds.0 10.252.24.25:6802/3144 -14884> 2013-04-03 10:39:20.931648 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 88 v187) v2 tid 17560) v2 -- ?+0 0x24f8240 con 0x208a840 -14883> 2013-04-03 10:39:21.384743 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 87 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21744 v187) v2 ==== 289+0+0 (632133321 0 0) 0x2cc5b80 con 0x200d420 -14882> 2013-04-03 10:39:21.384816 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21744 v187) v2 from mds.? 10.252.24.88:6800/17213 -14881> 2013-04-03 10:39:21.384828 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21744 v187) v2 -- ?+0 0x2ce2080 con 0x200d420 -14880> 2013-04-03 10:39:21.587863 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113455 ==== forward(pg_stats(1 pgs tid 29 v 605) v1) to leader v1 ==== 753+0+0 (1998960621 0 0) 0x2ce1dc0 con 0x208a840 -14879> 2013-04-03 10:39:21.587878 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -14878> 2013-04-03 10:39:21.587886 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -14877> 2013-04-03 10:39:21.587920 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 preprocess_query pg_stats(1 pgs tid 29 v 605) v1 from osd.1 10.252.24.25:6805/3572 -14876> 2013-04-03 10:39:21.587943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 prepare_update pg_stats(1 pgs tid 29 v 605) v1 from osd.1 10.252.24.25:6805/3572 -14875> 2013-04-03 10:39:21.587949 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 prepare_pg_stats pg_stats(1 pgs tid 29 v 605) v1 from osd.1 -14874> 2013-04-03 10:39:21.587955 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29038 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -14873> 2013-04-03 10:39:21.638141 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29038 encode_pending v 29039 -14872> 2013-04-03 10:39:21.638171 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427363) queue_proposal bl 650 bytes; ctx = 0x2c3d3c0 -14871> 2013-04-03 10:39:21.638178 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427363) propose_queued 427364 650 bytes -14870> 2013-04-03 10:39:21.639431 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427363 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787c80 -14869> 2013-04-03 10:39:21.689084 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113456 ==== paxos(accept lc 427363 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2286696947 0 0) 0x2787c80 con 0x208a840 -14868> 2013-04-03 10:39:21.689410 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427364 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787500 -14867> 2013-04-03 10:39:21.689437 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427364 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2ab4000 -14866> 2013-04-03 10:39:21.689506 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29038 update_from_paxos applying incremental 29039 -14865> 2013-04-03 10:39:21.689523 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 v29039: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1465B/s wr, 0op/s -14864> 2013-04-03 10:39:21.689543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 send_pg_creates to 0 pgs -14863> 2013-04-03 10:39:21.689548 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 update_logger -14862> 2013-04-03 10:39:21.689557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 create_pending v 29040 -14861> 2013-04-03 10:39:21.689566 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29039 _updated_stats for osd.1 10.252.24.25:6805/3572 -14860> 2013-04-03 10:39:21.689571 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 29) v1 tid 17561) v2 -- ?+0 0x24f8900 con 0x208a840 -14859> 2013-04-03 10:39:21.689581 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 check_osd_map already seen 605 >= 605 -14858> 2013-04-03 10:39:21.689584 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 update_logger -14857> 2013-04-03 10:39:21.689595 7fcdffcdb700 0 log [INF] : pgmap v29039: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1465B/s wr, 0op/s -14856> 2013-04-03 10:39:21.689596 7fcdffcdb700 10 send_log to self -14855> 2013-04-03 10:39:21.689597 7fcdffcdb700 10 log_queue is 1 last_log 105 sent 104 num 1 unsent 1 sending 1 -14854> 2013-04-03 10:39:21.689599 7fcdffcdb700 10 will send 2013-04-03 10:39:21.689596 mon.0 10.252.24.25:6789/0 105 : [INF] pgmap v29039: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1465B/s wr, 0op/s -14853> 2013-04-03 10:39:21.689611 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74600 -14852> 2013-04-03 10:39:21.689658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos -14851> 2013-04-03 10:39:21.689677 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 -14850> 2013-04-03 10:39:21.689695 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14849> 2013-04-03 10:39:21.689716 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74600 con 0x1bac160 -14848> 2013-04-03 10:39:21.689737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos -14847> 2013-04-03 10:39:21.689752 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 -14846> 2013-04-03 10:39:21.689761 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14845> 2013-04-03 10:39:21.689775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 preprocess_log log(1 entries) v1 from mon.0 -14844> 2013-04-03 10:39:21.689798 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14843> 2013-04-03 10:39:21.689810 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 prepare_log log(1 entries) v1 from mon.0 -14842> 2013-04-03 10:39:21.689828 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 logging 2013-04-03 10:39:21.689596 mon.0 10.252.24.25:6789/0 105 : [INF] pgmap v29039: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1465B/s wr, 0op/s -14841> 2013-04-03 10:39:21.715127 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113457 ==== paxos(lease_ack lc 427364 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (908279668 0 0) 0x2ab4000 con 0x208a840 -14840> 2013-04-03 10:39:21.715162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos -14839> 2013-04-03 10:39:21.715179 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394040 update_from_paxos version 394040 summary v 394040 -14838> 2013-04-03 10:39:21.715193 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14837> 2013-04-03 10:39:22.689286 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394040 encode_pending v394041 -14836> 2013-04-03 10:39:22.689322 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427364) queue_proposal bl 443 bytes; ctx = 0x2c3d100 -14835> 2013-04-03 10:39:22.689330 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427364) propose_queued 427365 443 bytes -14834> 2013-04-03 10:39:22.690434 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427364 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787280 -14833> 2013-04-03 10:39:22.716379 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113458 ==== paxos(accept lc 427364 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1481572528 0 0) 0x2787280 con 0x208a840 -14832> 2013-04-03 10:39:22.716693 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427365 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4000 -14831> 2013-04-03 10:39:22.716725 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427365 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2787c80 -14830> 2013-04-03 10:39:22.716760 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos -14829> 2013-04-03 10:39:22.716787 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394040 -14828> 2013-04-03 10:39:22.716801 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos latest full 393798 -14827> 2013-04-03 10:39:22.716819 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394041 update_from_paxos applying incremental log 394041 2013-04-03 10:39:21.689596 mon.0 10.252.24.25:6789/0 105 : [INF] pgmap v29039: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1465B/s wr, 0op/s -14826> 2013-04-03 10:39:22.716866 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 check_subs -14825> 2013-04-03 10:39:22.716898 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 create_pending v 394042 -14824> 2013-04-03 10:39:22.716920 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394041 _updated_log for mon.0 10.252.24.25:6789/0 -14823> 2013-04-03 10:39:22.716933 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 105) v1 -- ?+0 0x2224e00 con 0x1bac160 -14822> 2013-04-03 10:39:22.716986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos -14821> 2013-04-03 10:39:22.717002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 -14820> 2013-04-03 10:39:22.717020 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14819> 2013-04-03 10:39:22.717039 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 105) v1 ==== 0+0+0 (0 0 0) 0x2224e00 con 0x1bac160 -14818> 2013-04-03 10:39:22.717047 7fcdffcdb700 10 handle_log_ack log(last 105) v1 -14817> 2013-04-03 10:39:22.717049 7fcdffcdb700 10 logged 2013-04-03 10:39:21.689596 mon.0 10.252.24.25:6789/0 105 : [INF] pgmap v29039: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1465B/s wr, 0op/s -14816> 2013-04-03 10:39:22.728609 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113459 ==== paxos(lease_ack lc 427365 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (577297511 0 0) 0x2787c80 con 0x208a840 -14815> 2013-04-03 10:39:22.728644 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos -14814> 2013-04-03 10:39:22.728660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 -14813> 2013-04-03 10:39:22.728675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14812> 2013-04-03 10:39:23.346836 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29039 check_down_pgs -14811> 2013-04-03 10:39:23.346865 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29039 v29039: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1465B/s wr, 0op/s -14810> 2013-04-03 10:39:23.346892 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -14809> 2013-04-03 10:39:23.346916 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -14808> 2013-04-03 10:39:23.346953 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -14807> 2013-04-03 10:39:23.346984 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos -14806> 2013-04-03 10:39:23.347002 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 -14805> 2013-04-03 10:39:23.347012 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394041 log -14804> 2013-04-03 10:39:23.347029 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14803> 2013-04-03 10:39:23.347048 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -14802> 2013-04-03 10:39:23.585494 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 88 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140374 v187) v2 ==== 289+0+0 (1443209614 0 0) 0x2ce1b00 con 0x2118420 -14801> 2013-04-03 10:39:23.585547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140374 v187) v2 from mds.? 10.252.58.31:6800/10908 -14800> 2013-04-03 10:39:23.585558 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140374 v187) v2 -- ?+0 0x2ce1dc0 con 0x2118420 -14799> 2013-04-03 10:39:24.931746 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113460 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 89 v187) v2) to leader v1 ==== 560+0+0 (1611920486 0 0) 0x2a52b80 con 0x208a840 -14798> 2013-04-03 10:39:24.931766 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -14797> 2013-04-03 10:39:24.931776 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x295e340 from 10.252.24.88:6789/0 -14796> 2013-04-03 10:39:24.931828 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 89 v187) v2 from mds.0 10.252.24.25:6802/3144 -14795> 2013-04-03 10:39:24.931840 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 89 v187) v2 tid 17562) v2 -- ?+0 0x24f8240 con 0x208a840 -14794> 2013-04-03 10:39:25.384862 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 88 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21745 v187) v2 ==== 289+0+0 (113453492 0 0) 0x20e4340 con 0x200d420 -14793> 2013-04-03 10:39:25.384942 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21745 v187) v2 from mds.? 10.252.24.88:6800/17213 -14792> 2013-04-03 10:39:25.384955 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21745 v187) v2 -- ?+0 0x2a52b80 con 0x200d420 -14791> 2013-04-03 10:39:25.716799 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427365 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2787a00 -14790> 2013-04-03 10:39:25.717585 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113461 ==== paxos(lease_ack lc 427365 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1375877215 0 0) 0x2787a00 con 0x208a840 -14789> 2013-04-03 10:39:25.717633 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos -14788> 2013-04-03 10:39:25.717652 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 -14787> 2013-04-03 10:39:25.717669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14786> 2013-04-03 10:39:27.585667 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 89 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140375 v187) v2 ==== 289+0+0 (1969954547 0 0) 0x2ce1840 con 0x2118420 -14785> 2013-04-03 10:39:27.585740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140375 v187) v2 from mds.? 10.252.58.31:6800/10908 -14784> 2013-04-03 10:39:27.585753 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140375 v187) v2 -- ?+0 0x20e4340 con 0x2118420 -14783> 2013-04-03 10:39:28.227932 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113462 ==== forward(pg_stats(1 pgs tid 10 v 605) v1) to leader v1 ==== 757+0+0 (4047137701 0 0) 0x2ce1b00 con 0x208a840 -14782> 2013-04-03 10:39:28.227953 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -14781> 2013-04-03 10:39:28.227963 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -14780> 2013-04-03 10:39:28.228013 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 preprocess_query pg_stats(1 pgs tid 10 v 605) v1 from osd.2 10.252.24.30:6801/5656 -14779> 2013-04-03 10:39:28.228037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 prepare_update pg_stats(1 pgs tid 10 v 605) v1 from osd.2 10.252.24.30:6801/5656 -14778> 2013-04-03 10:39:28.228043 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 prepare_pg_stats pg_stats(1 pgs tid 10 v 605) v1 from osd.2 -14777> 2013-04-03 10:39:28.228050 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29039 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -14776> 2013-04-03 10:39:28.278205 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29039 encode_pending v 29040 -14775> 2013-04-03 10:39:28.278241 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427365) queue_proposal bl 654 bytes; ctx = 0x2c3d3b0 -14774> 2013-04-03 10:39:28.278250 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427365) propose_queued 427366 654 bytes -14773> 2013-04-03 10:39:28.279612 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427365 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2acd780 -14772> 2013-04-03 10:39:28.306840 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113463 ==== paxos(accept lc 427365 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3909733261 0 0) 0x2acd780 con 0x208a840 -14771> 2013-04-03 10:39:28.321357 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427366 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2d26c80 -14770> 2013-04-03 10:39:28.321386 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427366 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2d26a00 -14769> 2013-04-03 10:39:28.321460 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29039 update_from_paxos applying incremental 29040 -14768> 2013-04-03 10:39:28.321476 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29040 v29040: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -14767> 2013-04-03 10:39:28.321494 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29040 send_pg_creates to 0 pgs -14766> 2013-04-03 10:39:28.321497 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29040 update_logger -14765> 2013-04-03 10:39:28.321506 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29040 create_pending v 29041 -14764> 2013-04-03 10:39:28.321514 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29040 _updated_stats for osd.2 10.252.24.30:6801/5656 -14763> 2013-04-03 10:39:28.321520 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 10) v1 tid 17563) v2 -- ?+0 0x24f8900 con 0x208a840 -14762> 2013-04-03 10:39:28.321530 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29040 check_osd_map already seen 605 >= 605 -14761> 2013-04-03 10:39:28.321533 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29040 update_logger -14760> 2013-04-03 10:39:28.321543 7fcdffcdb700 0 log [INF] : pgmap v29040: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -14759> 2013-04-03 10:39:28.321545 7fcdffcdb700 10 send_log to self -14758> 2013-04-03 10:39:28.321545 7fcdffcdb700 10 log_queue is 1 last_log 106 sent 105 num 1 unsent 1 sending 1 -14757> 2013-04-03 10:39:28.321547 7fcdffcdb700 10 will send 2013-04-03 10:39:28.321544 mon.0 10.252.24.25:6789/0 106 : [INF] pgmap v29040: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -14756> 2013-04-03 10:39:28.321553 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74600 -14755> 2013-04-03 10:39:28.321603 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos -14754> 2013-04-03 10:39:28.321622 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 -14753> 2013-04-03 10:39:28.321640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14752> 2013-04-03 10:39:28.321661 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74600 con 0x1bac160 -14751> 2013-04-03 10:39:28.321683 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos -14750> 2013-04-03 10:39:28.321699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 -14749> 2013-04-03 10:39:28.321708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14748> 2013-04-03 10:39:28.321721 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 preprocess_log log(1 entries) v1 from mon.0 -14747> 2013-04-03 10:39:28.321743 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14746> 2013-04-03 10:39:28.321755 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 prepare_log log(1 entries) v1 from mon.0 -14745> 2013-04-03 10:39:28.321766 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 logging 2013-04-03 10:39:28.321544 mon.0 10.252.24.25:6789/0 106 : [INF] pgmap v29040: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -14744> 2013-04-03 10:39:28.347163 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29040 check_down_pgs -14743> 2013-04-03 10:39:28.347199 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29040 v29040: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -14742> 2013-04-03 10:39:28.347222 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -14741> 2013-04-03 10:39:28.347243 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -14740> 2013-04-03 10:39:28.347279 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -14739> 2013-04-03 10:39:28.347309 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos -14738> 2013-04-03 10:39:28.347325 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 -14737> 2013-04-03 10:39:28.347334 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394041 log -14736> 2013-04-03 10:39:28.347350 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14735> 2013-04-03 10:39:28.347366 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -14734> 2013-04-03 10:39:28.349569 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113464 ==== paxos(lease_ack lc 427366 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2104085578 0 0) 0x2d26a00 con 0x208a840 -14733> 2013-04-03 10:39:28.349611 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos -14732> 2013-04-03 10:39:28.349627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394041 update_from_paxos version 394041 summary v 394041 -14731> 2013-04-03 10:39:28.349643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -14730> 2013-04-03 10:39:28.485059 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113465 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (424405211 0 0) 0x295d580 con 0x208a840 -14729> 2013-04-03 10:39:28.485071 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -14728> 2013-04-03 10:39:28.485076 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -14727> 2013-04-03 10:39:28.485083 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 -14726> 2013-04-03 10:39:28.485086 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( start_reply flags( reply-to(mon.2 10.252.24.88:6789/0) ) ) v1 -- ?+0 0x2ce1b00 -14725> 2013-04-03 10:39:28.931817 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113466 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 90 v187) v2) to leader v1 ==== 560+0+0 (39882088 0 0) 0x2a83b00 con 0x208a840 -14724> 2013-04-03 10:39:28.931832 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -14723> 2013-04-03 10:39:28.931841 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83580 from 10.252.24.88:6789/0 -14722> 2013-04-03 10:39:28.931879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 90 v187) v2 from mds.0 10.252.24.25:6802/3144 -14721> 2013-04-03 10:39:28.931891 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 90 v187) v2 tid 17564) v2 -- ?+0 0x24f8240 con 0x208a840 -14720> 2013-04-03 10:39:29.307047 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394041 encode_pending v394042 -14719> 2013-04-03 10:39:29.307078 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427366) queue_proposal bl 443 bytes; ctx = 0x2c3d010 -14718> 2013-04-03 10:39:29.307085 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427366) propose_queued 427367 443 bytes -14717> 2013-04-03 10:39:29.308030 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427366 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2acd500 -14716> 2013-04-03 10:39:29.332977 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113467 ==== paxos(accept lc 427366 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3950724017 0 0) 0x2acd500 con 0x208a840 -14715> 2013-04-03 10:39:29.338604 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427367 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2d26a00 -14714> 2013-04-03 10:39:29.338638 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427367 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2acd780 -14713> 2013-04-03 10:39:29.338680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos -14712> 2013-04-03 10:39:29.338707 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos version 394042 summary v 394041 -14711> 2013-04-03 10:39:29.338722 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos latest full 393798 -14710> 2013-04-03 10:39:29.338742 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos applying incremental log 394042 2013-04-03 10:39:28.321544 mon.0 10.252.24.25:6789/0 106 : [INF] pgmap v29040: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -14709> 2013-04-03 10:39:29.338789 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 check_subs -14708> 2013-04-03 10:39:29.338821 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 create_pending v 394043 -14707> 2013-04-03 10:39:29.338844 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 _updated_log for mon.0 10.252.24.25:6789/0 -14706> 2013-04-03 10:39:29.338858 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 106) v1 -- ?+0 0x22248c0 con 0x1bac160 -14705> 2013-04-03 10:39:29.338909 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos -14704> 2013-04-03 10:39:29.338925 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos version 394042 summary v 394042 -14703> 2013-04-03 10:39:29.338944 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14702> 2013-04-03 10:39:29.338967 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 106) v1 ==== 0+0+0 (0 0 0) 0x22248c0 con 0x1bac160 -14701> 2013-04-03 10:39:29.338975 7fcdffcdb700 10 handle_log_ack log(last 106) v1 -14700> 2013-04-03 10:39:29.338977 7fcdffcdb700 10 logged 2013-04-03 10:39:28.321544 mon.0 10.252.24.25:6789/0 106 : [INF] pgmap v29040: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -14699> 2013-04-03 10:39:29.353653 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113468 ==== paxos(lease_ack lc 427367 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3730250084 0 0) 0x2acd780 con 0x208a840 -14698> 2013-04-03 10:39:29.353692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos -14697> 2013-04-03 10:39:29.353709 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos version 394042 summary v 394042 -14696> 2013-04-03 10:39:29.353725 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14695> 2013-04-03 10:39:29.384956 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 89 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21746 v187) v2 ==== 289+0+0 (1668343859 0 0) 0x29c58c0 con 0x200d420 -14694> 2013-04-03 10:39:29.385013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21746 v187) v2 from mds.? 10.252.24.88:6800/17213 -14693> 2013-04-03 10:39:29.385026 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21746 v187) v2 -- ?+0 0x2a83b00 con 0x200d420 -14692> 2013-04-03 10:39:31.585786 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 90 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140376 v187) v2 ==== 289+0+0 (2214309785 0 0) 0x2ce1580 con 0x2118420 -14691> 2013-04-03 10:39:31.585859 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140376 v187) v2 from mds.? 10.252.58.31:6800/10908 -14690> 2013-04-03 10:39:31.585873 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140376 v187) v2 -- ?+0 0x29c58c0 con 0x2118420 -14689> 2013-04-03 10:39:31.588938 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113469 ==== forward(pg_stats(1 pgs tid 30 v 605) v1) to leader v1 ==== 753+0+0 (1774420798 0 0) 0x295d580 con 0x208a840 -14688> 2013-04-03 10:39:31.588953 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -14687> 2013-04-03 10:39:31.588962 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -14686> 2013-04-03 10:39:31.588999 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29040 preprocess_query pg_stats(1 pgs tid 30 v 605) v1 from osd.1 10.252.24.25:6805/3572 -14685> 2013-04-03 10:39:31.589024 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29040 prepare_update pg_stats(1 pgs tid 30 v 605) v1 from osd.1 10.252.24.25:6805/3572 -14684> 2013-04-03 10:39:31.589030 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29040 prepare_pg_stats pg_stats(1 pgs tid 30 v 605) v1 from osd.1 -14683> 2013-04-03 10:39:31.589037 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29040 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -14682> 2013-04-03 10:39:31.639218 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29040 encode_pending v 29041 -14681> 2013-04-03 10:39:31.639259 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427367) queue_proposal bl 650 bytes; ctx = 0x2c3d0e0 -14680> 2013-04-03 10:39:31.639268 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427367) propose_queued 427368 650 bytes -14679> 2013-04-03 10:39:31.640358 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427367 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2acdc80 -14678> 2013-04-03 10:39:31.694214 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113470 ==== paxos(accept lc 427367 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2901557322 0 0) 0x2acdc80 con 0x208a840 -14677> 2013-04-03 10:39:31.695046 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427368 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2acd780 -14676> 2013-04-03 10:39:31.695079 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427368 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2acd500 -14675> 2013-04-03 10:39:31.695153 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29040 update_from_paxos applying incremental 29041 -14674> 2013-04-03 10:39:31.695170 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 v29041: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 4197B/s wr, 0op/s -14673> 2013-04-03 10:39:31.695193 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 send_pg_creates to 0 pgs -14672> 2013-04-03 10:39:31.695198 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 update_logger -14671> 2013-04-03 10:39:31.695208 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 create_pending v 29042 -14670> 2013-04-03 10:39:31.695217 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 _updated_stats for osd.1 10.252.24.25:6805/3572 -14669> 2013-04-03 10:39:31.695224 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30) v1 tid 17565) v2 -- ?+0 0x24f8900 con 0x208a840 -14668> 2013-04-03 10:39:31.695233 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 check_osd_map already seen 605 >= 605 -14667> 2013-04-03 10:39:31.695237 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 update_logger -14666> 2013-04-03 10:39:31.695248 7fcdffcdb700 0 log [INF] : pgmap v29041: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 4197B/s wr, 0op/s -14665> 2013-04-03 10:39:31.695249 7fcdffcdb700 10 send_log to self -14664> 2013-04-03 10:39:31.695250 7fcdffcdb700 10 log_queue is 1 last_log 107 sent 106 num 1 unsent 1 sending 1 -14663> 2013-04-03 10:39:31.695252 7fcdffcdb700 10 will send 2013-04-03 10:39:31.695249 mon.0 10.252.24.25:6789/0 107 : [INF] pgmap v29041: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 4197B/s wr, 0op/s -14662> 2013-04-03 10:39:31.695259 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74600 -14661> 2013-04-03 10:39:31.695305 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos -14660> 2013-04-03 10:39:31.695324 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos version 394042 summary v 394042 -14659> 2013-04-03 10:39:31.695344 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14658> 2013-04-03 10:39:31.695365 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74600 con 0x1bac160 -14657> 2013-04-03 10:39:31.695387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos -14656> 2013-04-03 10:39:31.695405 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos version 394042 summary v 394042 -14655> 2013-04-03 10:39:31.695414 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14654> 2013-04-03 10:39:31.695427 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 preprocess_log log(1 entries) v1 from mon.0 -14653> 2013-04-03 10:39:31.695451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14652> 2013-04-03 10:39:31.695463 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 prepare_log log(1 entries) v1 from mon.0 -14651> 2013-04-03 10:39:31.695473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 logging 2013-04-03 10:39:31.695249 mon.0 10.252.24.25:6789/0 107 : [INF] pgmap v29041: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 4197B/s wr, 0op/s -14650> 2013-04-03 10:39:31.707885 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113471 ==== paxos(lease_ack lc 427368 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3790551543 0 0) 0x2acd500 con 0x208a840 -14649> 2013-04-03 10:39:31.707933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos -14648> 2013-04-03 10:39:31.707951 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 update_from_paxos version 394042 summary v 394042 -14647> 2013-04-03 10:39:31.707967 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14646> 2013-04-03 10:39:31.754340 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113472 ==== forward(pg_stats(2 pgs tid 30446 v 605) v1) to leader v1 ==== 1130+0+0 (974326940 0 0) 0x2a84600 con 0x208a840 -14645> 2013-04-03 10:39:31.754354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -14644> 2013-04-03 10:39:31.754362 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -14643> 2013-04-03 10:39:31.754392 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 preprocess_query pg_stats(2 pgs tid 30446 v 605) v1 from osd.3 10.252.24.88:6801/17538 -14642> 2013-04-03 10:39:31.754413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 prepare_update pg_stats(2 pgs tid 30446 v 605) v1 from osd.3 10.252.24.88:6801/17538 -14641> 2013-04-03 10:39:31.754418 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 prepare_pg_stats pg_stats(2 pgs tid 30446 v 605) v1 from osd.3 -14640> 2013-04-03 10:39:31.754422 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 got osd.3 osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -14639> 2013-04-03 10:39:32.694411 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 encode_pending v 29042 -14638> 2013-04-03 10:39:32.694449 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427368) queue_proposal bl 1027 bytes; ctx = 0x2c3d3d0 -14637> 2013-04-03 10:39:32.694460 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427368) propose_queued 427369 1027 bytes -14636> 2013-04-03 10:39:32.695553 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427368 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787a00 -14635> 2013-04-03 10:39:32.695655 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394042 encode_pending v394043 -14634> 2013-04-03 10:39:32.695683 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427326..427368) queue_proposal bl 443 bytes; ctx = 0x2c3d330 -14633> 2013-04-03 10:39:32.695690 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427326..427368) propose_new_value not active; proposal queued -14632> 2013-04-03 10:39:32.724649 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113473 ==== paxos(accept lc 427368 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4004451418 0 0) 0x2787a00 con 0x208a840 -14631> 2013-04-03 10:39:32.724944 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427369 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2acd500 -14630> 2013-04-03 10:39:32.724977 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427369 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2acdc80 -14629> 2013-04-03 10:39:32.725052 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29041 update_from_paxos applying incremental 29042 -14628> 2013-04-03 10:39:32.725075 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 v29042: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 79532B/s wr, 1op/s -14627> 2013-04-03 10:39:32.725096 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 send_pg_creates to 0 pgs -14626> 2013-04-03 10:39:32.725100 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 update_logger -14625> 2013-04-03 10:39:32.725114 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 create_pending v 29043 -14624> 2013-04-03 10:39:32.725125 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 _updated_stats for osd.3 10.252.24.88:6801/17538 -14623> 2013-04-03 10:39:32.725132 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(2 pgs tid 30446) v1 tid 17566) v2 -- ?+0 0x24f8240 con 0x208a840 -14622> 2013-04-03 10:39:32.725143 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 check_osd_map already seen 605 >= 605 -14621> 2013-04-03 10:39:32.725146 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 update_logger -14620> 2013-04-03 10:39:32.725157 7fcdffcdb700 0 log [INF] : pgmap v29042: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 79532B/s wr, 1op/s -14619> 2013-04-03 10:39:32.725158 7fcdffcdb700 10 send_log to self -14618> 2013-04-03 10:39:32.725159 7fcdffcdb700 10 log_queue is 2 last_log 108 sent 107 num 2 unsent 1 sending 1 -14617> 2013-04-03 10:39:32.725161 7fcdffcdb700 10 will send 2013-04-03 10:39:31.695249 mon.0 10.252.24.25:6789/0 107 : [INF] pgmap v29041: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 4197B/s wr, 0op/s -14616> 2013-04-03 10:39:32.725168 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75c00 -14615> 2013-04-03 10:39:32.725194 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427369) propose_queued 427370 443 bytes -14614> 2013-04-03 10:39:32.725409 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427369 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2d26780 -14613> 2013-04-03 10:39:32.725443 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75c00 con 0x1bac160 -14612> 2013-04-03 10:39:32.761926 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113474 ==== paxos(lease_ack lc 427369 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2874542247 0 0) 0x2d26780 con 0x208a840 -14611> 2013-04-03 10:39:32.901279 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113475 ==== paxos(accept lc 427369 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1834560228 0 0) 0x2acdc80 con 0x208a840 -14610> 2013-04-03 10:39:32.901506 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427370 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2d26780 -14609> 2013-04-03 10:39:32.901539 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427370 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2787a00 -14608> 2013-04-03 10:39:32.901577 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos -14607> 2013-04-03 10:39:32.901604 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394042 -14606> 2013-04-03 10:39:32.901618 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos latest full 393798 -14605> 2013-04-03 10:39:32.901637 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos applying incremental log 394043 2013-04-03 10:39:31.695249 mon.0 10.252.24.25:6789/0 107 : [INF] pgmap v29041: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 4197B/s wr, 0op/s -14604> 2013-04-03 10:39:32.901684 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 check_subs -14603> 2013-04-03 10:39:32.901717 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 create_pending v 394044 -14602> 2013-04-03 10:39:32.901740 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 _updated_log for mon.0 10.252.24.25:6789/0 -14601> 2013-04-03 10:39:32.901753 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 107) v1 -- ?+0 0x2224540 con 0x1bac160 -14600> 2013-04-03 10:39:32.901773 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos -14599> 2013-04-03 10:39:32.901789 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394043 -14598> 2013-04-03 10:39:32.901798 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14597> 2013-04-03 10:39:32.901818 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 preprocess_log log(1 entries) v1 from mon.0 -14596> 2013-04-03 10:39:32.901839 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 nothing new -14595> 2013-04-03 10:39:32.901889 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos -14594> 2013-04-03 10:39:32.901905 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394043 -14593> 2013-04-03 10:39:32.901923 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14592> 2013-04-03 10:39:32.901947 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 107) v1 ==== 0+0+0 (0 0 0) 0x2224540 con 0x1bac160 -14591> 2013-04-03 10:39:32.901956 7fcdffcdb700 10 handle_log_ack log(last 107) v1 -14590> 2013-04-03 10:39:32.901957 7fcdffcdb700 10 logged 2013-04-03 10:39:31.695249 mon.0 10.252.24.25:6789/0 107 : [INF] pgmap v29041: 240 pgs: 240 active+clean; 253 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 4197B/s wr, 0op/s -14589> 2013-04-03 10:39:33.018313 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113476 ==== paxos(lease_ack lc 427370 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (249814348 0 0) 0x2787a00 con 0x208a840 -14588> 2013-04-03 10:39:33.018354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos -14587> 2013-04-03 10:39:33.018371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394043 -14586> 2013-04-03 10:39:33.018387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14585> 2013-04-03 10:39:33.018655 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113477 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 91 v187) v2) to leader v1 ==== 560+0+0 (2466365711 0 0) 0x2a84340 con 0x208a840 -14584> 2013-04-03 10:39:33.018666 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -14583> 2013-04-03 10:39:33.018674 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a84b80 from 10.252.24.88:6789/0 -14582> 2013-04-03 10:39:33.018700 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 91 v187) v2 from mds.0 10.252.24.25:6802/3144 -14581> 2013-04-03 10:39:33.018711 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 91 v187) v2 tid 17567) v2 -- ?+0 0x24f8900 con 0x208a840 -14580> 2013-04-03 10:39:33.347509 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 check_down_pgs -14579> 2013-04-03 10:39:33.347545 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 v29042: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 79532B/s wr, 1op/s -14578> 2013-04-03 10:39:33.347573 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -14577> 2013-04-03 10:39:33.347601 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -14576> 2013-04-03 10:39:33.347639 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -14575> 2013-04-03 10:39:33.347671 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos -14574> 2013-04-03 10:39:33.347690 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394043 -14573> 2013-04-03 10:39:33.347700 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 log -14572> 2013-04-03 10:39:33.347718 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14571> 2013-04-03 10:39:33.347738 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -14570> 2013-04-03 10:39:33.385107 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 90 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21747 v187) v2 ==== 289+0+0 (1075730254 0 0) 0x29c4b00 con 0x200d420 -14569> 2013-04-03 10:39:33.385149 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21747 v187) v2 from mds.? 10.252.24.88:6800/17213 -14568> 2013-04-03 10:39:33.385160 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21747 v187) v2 -- ?+0 0x2a84340 con 0x200d420 -14567> 2013-04-03 10:39:35.462664 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113478 ==== forward(pg_stats(0 pgs tid 60 v 605) v1) to leader v1 ==== 368+0+0 (1997241021 0 0) 0x2a84600 con 0x208a840 -14566> 2013-04-03 10:39:35.462686 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 -14565> 2013-04-03 10:39:35.462696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -14564> 2013-04-03 10:39:35.462749 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 preprocess_query pg_stats(0 pgs tid 60 v 605) v1 from osd.4 10.252.24.32:6800/13473 -14563> 2013-04-03 10:39:35.462778 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 prepare_update pg_stats(0 pgs tid 60 v 605) v1 from osd.4 10.252.24.32:6800/13473 -14562> 2013-04-03 10:39:35.462784 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 prepare_pg_stats pg_stats(0 pgs tid 60 v 605) v1 from osd.4 -14561> 2013-04-03 10:39:35.462791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 got osd.4 osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[])) -14560> 2013-04-03 10:39:35.512968 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 encode_pending v 29043 -14559> 2013-04-03 10:39:35.513005 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427370) queue_proposal bl 265 bytes; ctx = 0x2c3d110 -14558> 2013-04-03 10:39:35.513017 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427370) propose_queued 427371 265 bytes -14557> 2013-04-03 10:39:35.514102 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427370 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787c80 -14556> 2013-04-03 10:39:35.536379 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113479 ==== paxos(accept lc 427370 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3347854519 0 0) 0x2787c80 con 0x208a840 -14555> 2013-04-03 10:39:35.537418 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427371 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787a00 -14554> 2013-04-03 10:39:35.537445 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427371 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2acdc80 -14553> 2013-04-03 10:39:35.537516 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29042 update_from_paxos applying incremental 29043 -14552> 2013-04-03 10:39:35.537527 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 v29043: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 83003B/s wr, 0op/s -14551> 2013-04-03 10:39:35.537545 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 send_pg_creates to 0 pgs -14550> 2013-04-03 10:39:35.537549 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 update_logger -14549> 2013-04-03 10:39:35.537559 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 create_pending v 29044 -14548> 2013-04-03 10:39:35.537569 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 _updated_stats for osd.4 10.252.24.32:6800/13473 -14547> 2013-04-03 10:39:35.537575 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(0 pgs tid 60) v1 tid 17568) v2 -- ?+0 0x24f8240 con 0x208a840 -14546> 2013-04-03 10:39:35.537584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 check_osd_map already seen 605 >= 605 -14545> 2013-04-03 10:39:35.537588 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 update_logger -14544> 2013-04-03 10:39:35.537598 7fcdffcdb700 0 log [INF] : pgmap v29043: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 83003B/s wr, 0op/s -14543> 2013-04-03 10:39:35.537600 7fcdffcdb700 10 send_log to self -14542> 2013-04-03 10:39:35.537601 7fcdffcdb700 10 log_queue is 2 last_log 109 sent 107 num 2 unsent 2 sending 2 -14541> 2013-04-03 10:39:35.537603 7fcdffcdb700 10 will send 2013-04-03 10:39:32.725158 mon.0 10.252.24.25:6789/0 108 : [INF] pgmap v29042: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 79532B/s wr, 1op/s -14540> 2013-04-03 10:39:35.537616 7fcdffcdb700 10 will send 2013-04-03 10:39:35.537600 mon.0 10.252.24.25:6789/0 109 : [INF] pgmap v29043: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 83003B/s wr, 0op/s -14539> 2013-04-03 10:39:35.537621 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b75c00 -14538> 2013-04-03 10:39:35.537672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos -14537> 2013-04-03 10:39:35.537692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394043 -14536> 2013-04-03 10:39:35.537712 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14535> 2013-04-03 10:39:35.537733 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75c00 con 0x1bac160 -14534> 2013-04-03 10:39:35.537755 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos -14533> 2013-04-03 10:39:35.537770 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394043 -14532> 2013-04-03 10:39:35.537779 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -14531> 2013-04-03 10:39:35.537791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 preprocess_log log(2 entries) v1 from mon.0 -14530> 2013-04-03 10:39:35.537831 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -14529> 2013-04-03 10:39:35.537846 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 prepare_log log(2 entries) v1 from mon.0 -14528> 2013-04-03 10:39:35.537858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 logging 2013-04-03 10:39:32.725158 mon.0 10.252.24.25:6789/0 108 : [INF] pgmap v29042: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 79532B/s wr, 1op/s -14527> 2013-04-03 10:39:35.537881 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 logging 2013-04-03 10:39:35.537600 mon.0 10.252.24.25:6789/0 109 : [INF] pgmap v29043: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 83003B/s wr, 0op/s -14526> 2013-04-03 10:39:35.550087 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113480 ==== paxos(lease_ack lc 427371 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (4029956713 0 0) 0x2acdc80 con 0x208a840 -14525> 2013-04-03 10:39:35.550124 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos -14524> 2013-04-03 10:39:35.550140 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 update_from_paxos version 394043 summary v 394043 -14523> 2013-04-03 10:39:35.550158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14522> 2013-04-03 10:39:35.585872 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 91 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140377 v187) v2 ==== 289+0+0 (2694123748 0 0) 0x2ce12c0 con 0x2118420 -14521> 2013-04-03 10:39:35.585929 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140377 v187) v2 from mds.? 10.252.58.31:6800/10908 -14520> 2013-04-03 10:39:35.585941 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140377 v187) v2 -- ?+0 0x2a84600 con 0x2118420 -14519> 2013-04-03 10:39:36.536579 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394043 encode_pending v394044 -14518> 2013-04-03 10:39:36.536609 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427371) queue_proposal bl 728 bytes; ctx = 0x2c3d000 -14517> 2013-04-03 10:39:36.536617 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427371) propose_queued 427372 728 bytes -14516> 2013-04-03 10:39:36.537743 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427371 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787280 -14515> 2013-04-03 10:39:36.575187 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113481 ==== paxos(accept lc 427371 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3707463290 0 0) 0x2787280 con 0x208a840 -14514> 2013-04-03 10:39:36.575492 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427372 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2acdc80 -14513> 2013-04-03 10:39:36.575525 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427372 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2787c80 -14512> 2013-04-03 10:39:36.575565 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos -14511> 2013-04-03 10:39:36.575593 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos version 394044 summary v 394043 -14510> 2013-04-03 10:39:36.575607 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos latest full 393798 -14509> 2013-04-03 10:39:36.575625 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos applying incremental log 394044 2013-04-03 10:39:32.725158 mon.0 10.252.24.25:6789/0 108 : [INF] pgmap v29042: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 79532B/s wr, 1op/s -14508> 2013-04-03 10:39:36.575654 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos applying incremental log 394044 2013-04-03 10:39:35.537600 mon.0 10.252.24.25:6789/0 109 : [INF] pgmap v29043: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 83003B/s wr, 0op/s -14507> 2013-04-03 10:39:36.575694 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 check_subs -14506> 2013-04-03 10:39:36.575726 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 create_pending v 394045 -14505> 2013-04-03 10:39:36.575749 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 _updated_log for mon.0 10.252.24.25:6789/0 -14504> 2013-04-03 10:39:36.575762 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 109) v1 -- ?+0 0x2224c40 con 0x1bac160 -14503> 2013-04-03 10:39:36.575814 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos -14502> 2013-04-03 10:39:36.575831 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos version 394044 summary v 394044 -14501> 2013-04-03 10:39:36.575849 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14500> 2013-04-03 10:39:36.575871 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 109) v1 ==== 0+0+0 (0 0 0) 0x2224c40 con 0x1bac160 -14499> 2013-04-03 10:39:36.575879 7fcdffcdb700 10 handle_log_ack log(last 109) v1 -14498> 2013-04-03 10:39:36.575884 7fcdffcdb700 10 logged 2013-04-03 10:39:32.725158 mon.0 10.252.24.25:6789/0 108 : [INF] pgmap v29042: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 79532B/s wr, 1op/s -14497> 2013-04-03 10:39:36.575891 7fcdffcdb700 10 logged 2013-04-03 10:39:35.537600 mon.0 10.252.24.25:6789/0 109 : [INF] pgmap v29043: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 83003B/s wr, 0op/s -14496> 2013-04-03 10:39:36.587556 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113482 ==== paxos(lease_ack lc 427372 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (958702040 0 0) 0x2787c80 con 0x208a840 -14495> 2013-04-03 10:39:36.587592 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos -14494> 2013-04-03 10:39:36.587609 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos version 394044 summary v 394044 -14493> 2013-04-03 10:39:36.587626 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14492> 2013-04-03 10:39:36.589507 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113483 ==== forward(pg_stats(1 pgs tid 31 v 605) v1) to leader v1 ==== 753+0+0 (1573218156 0 0) 0x2a83dc0 con 0x208a840 -14491> 2013-04-03 10:39:36.589519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -14490> 2013-04-03 10:39:36.589528 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -14489> 2013-04-03 10:39:36.589557 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 preprocess_query pg_stats(1 pgs tid 31 v 605) v1 from osd.1 10.252.24.25:6805/3572 -14488> 2013-04-03 10:39:36.589579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 prepare_update pg_stats(1 pgs tid 31 v 605) v1 from osd.1 10.252.24.25:6805/3572 -14487> 2013-04-03 10:39:36.589585 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 prepare_pg_stats pg_stats(1 pgs tid 31 v 605) v1 from osd.1 -14486> 2013-04-03 10:39:36.589592 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -14485> 2013-04-03 10:39:36.818125 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113484 ==== forward(pg_stats(0 pgs tid 409 v 605) v1) to leader v1 ==== 384+0+0 (2856549685 0 0) 0x2a848c0 con 0x208a840 -14484> 2013-04-03 10:39:36.818138 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -14483> 2013-04-03 10:39:36.818147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -14482> 2013-04-03 10:39:36.818178 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 preprocess_query pg_stats(0 pgs tid 409 v 605) v1 from osd.0 10.252.58.31:6802/2682 -14481> 2013-04-03 10:39:36.818199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 prepare_update pg_stats(0 pgs tid 409 v 605) v1 from osd.0 10.252.58.31:6802/2682 -14480> 2013-04-03 10:39:36.818205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 prepare_pg_stats pg_stats(0 pgs tid 409 v 605) v1 from osd.0 -14479> 2013-04-03 10:39:36.818211 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 got osd.0 osd_stat(1139 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1139 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -14478> 2013-04-03 10:39:36.932104 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113485 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 92 v187) v2) to leader v1 ==== 560+0+0 (1816726527 0 0) 0x2a83840 con 0x208a840 -14477> 2013-04-03 10:39:36.932118 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -14476> 2013-04-03 10:39:36.932126 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83000 from 10.252.24.88:6789/0 -14475> 2013-04-03 10:39:36.932157 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 92 v187) v2 from mds.0 10.252.24.25:6802/3144 -14474> 2013-04-03 10:39:36.932168 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 92 v187) v2 tid 17571) v2 -- ?+0 0x2b26b40 con 0x208a840 -14473> 2013-04-03 10:39:37.385209 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 91 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21748 v187) v2 ==== 289+0+0 (2820107069 0 0) 0x29c5340 con 0x200d420 -14472> 2013-04-03 10:39:37.385286 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21748 v187) v2 from mds.? 10.252.24.88:6800/17213 -14471> 2013-04-03 10:39:37.385301 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21748 v187) v2 -- ?+0 0x2a83840 con 0x200d420 -14470> 2013-04-03 10:39:37.575348 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 encode_pending v 29044 -14469> 2013-04-03 10:39:37.575382 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427372) queue_proposal bl 716 bytes; ctx = 0x2c3d3a0 -14468> 2013-04-03 10:39:37.575391 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427372) propose_queued 427373 716 bytes -14467> 2013-04-03 10:39:37.577049 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427372 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787780 -14466> 2013-04-03 10:39:37.612626 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113486 ==== paxos(accept lc 427372 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1242818463 0 0) 0x2787780 con 0x208a840 -14465> 2013-04-03 10:39:37.613526 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427373 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787c80 -14464> 2013-04-03 10:39:37.613556 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427373 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2787280 -14463> 2013-04-03 10:39:37.613632 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29043 update_from_paxos applying incremental 29044 -14462> 2013-04-03 10:39:37.613654 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 v29044: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s -14461> 2013-04-03 10:39:37.613674 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 send_pg_creates to 0 pgs -14460> 2013-04-03 10:39:37.613679 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 update_logger -14459> 2013-04-03 10:39:37.613690 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 create_pending v 29045 -14458> 2013-04-03 10:39:37.613699 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 _updated_stats for osd.1 10.252.24.25:6805/3572 -14457> 2013-04-03 10:39:37.613706 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 31) v1 tid 17569) v2 -- ?+0 0x24f8900 con 0x208a840 -14456> 2013-04-03 10:39:37.613719 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 _updated_stats for osd.0 10.252.58.31:6802/2682 -14455> 2013-04-03 10:39:37.613724 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(0 pgs tid 409) v1 tid 17570) v2 -- ?+0 0x24f8240 con 0x208a840 -14454> 2013-04-03 10:39:37.613758 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 check_osd_map already seen 605 >= 605 -14453> 2013-04-03 10:39:37.613763 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 update_logger -14452> 2013-04-03 10:39:37.613774 7fcdffcdb700 0 log [INF] : pgmap v29044: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s -14451> 2013-04-03 10:39:37.613775 7fcdffcdb700 10 send_log to self -14450> 2013-04-03 10:39:37.613776 7fcdffcdb700 10 log_queue is 1 last_log 110 sent 109 num 1 unsent 1 sending 1 -14449> 2013-04-03 10:39:37.613778 7fcdffcdb700 10 will send 2013-04-03 10:39:37.613775 mon.0 10.252.24.25:6789/0 110 : [INF] pgmap v29044: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s -14448> 2013-04-03 10:39:37.613785 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75a00 -14447> 2013-04-03 10:39:37.613843 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos -14446> 2013-04-03 10:39:37.613863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos version 394044 summary v 394044 -14445> 2013-04-03 10:39:37.613882 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14444> 2013-04-03 10:39:37.613904 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75a00 con 0x1bac160 -14443> 2013-04-03 10:39:37.613926 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos -14442> 2013-04-03 10:39:37.613941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos version 394044 summary v 394044 -14441> 2013-04-03 10:39:37.613950 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14440> 2013-04-03 10:39:37.613962 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 preprocess_log log(1 entries) v1 from mon.0 -14439> 2013-04-03 10:39:37.613985 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14438> 2013-04-03 10:39:37.613998 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 prepare_log log(1 entries) v1 from mon.0 -14437> 2013-04-03 10:39:37.614013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 logging 2013-04-03 10:39:37.613775 mon.0 10.252.24.25:6789/0 110 : [INF] pgmap v29044: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s -14436> 2013-04-03 10:39:37.626281 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113487 ==== paxos(lease_ack lc 427373 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3387571707 0 0) 0x2787280 con 0x208a840 -14435> 2013-04-03 10:39:37.626317 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos -14434> 2013-04-03 10:39:37.626333 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos version 394044 summary v 394044 -14433> 2013-04-03 10:39:37.626348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14432> 2013-04-03 10:39:38.347887 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 check_down_pgs -14431> 2013-04-03 10:39:38.347917 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 v29044: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s -14430> 2013-04-03 10:39:38.347946 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -14429> 2013-04-03 10:39:38.347971 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -14428> 2013-04-03 10:39:38.348008 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -14427> 2013-04-03 10:39:38.348040 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos -14426> 2013-04-03 10:39:38.348064 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 update_from_paxos version 394044 summary v 394044 -14425> 2013-04-03 10:39:38.348074 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 log -14424> 2013-04-03 10:39:38.348091 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14423> 2013-04-03 10:39:38.348110 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -14422> 2013-04-03 10:39:38.612805 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394044 encode_pending v394045 -14421> 2013-04-03 10:39:38.612831 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427373) queue_proposal bl 443 bytes; ctx = 0x2c3d0d0 -14420> 2013-04-03 10:39:38.612840 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427373) propose_queued 427374 443 bytes -14419> 2013-04-03 10:39:38.613911 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427373 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4780 -14418> 2013-04-03 10:39:38.638635 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113488 ==== paxos(accept lc 427373 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2480093304 0 0) 0x2ab4780 con 0x208a840 -14417> 2013-04-03 10:39:38.646074 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427374 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787280 -14416> 2013-04-03 10:39:38.646107 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427374 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2787780 -14415> 2013-04-03 10:39:38.646147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos -14414> 2013-04-03 10:39:38.646174 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos version 394045 summary v 394044 -14413> 2013-04-03 10:39:38.646189 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos latest full 393798 -14412> 2013-04-03 10:39:38.646208 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos applying incremental log 394045 2013-04-03 10:39:37.613775 mon.0 10.252.24.25:6789/0 110 : [INF] pgmap v29044: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s -14411> 2013-04-03 10:39:38.646254 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 check_subs -14410> 2013-04-03 10:39:38.646287 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 create_pending v 394046 -14409> 2013-04-03 10:39:38.646310 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 _updated_log for mon.0 10.252.24.25:6789/0 -14408> 2013-04-03 10:39:38.646323 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 110) v1 -- ?+0 0x2224380 con 0x1bac160 -14407> 2013-04-03 10:39:38.646369 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos -14406> 2013-04-03 10:39:38.646385 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos version 394045 summary v 394045 -14405> 2013-04-03 10:39:38.646402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14404> 2013-04-03 10:39:38.646427 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 110) v1 ==== 0+0+0 (0 0 0) 0x2224380 con 0x1bac160 -14403> 2013-04-03 10:39:38.646435 7fcdffcdb700 10 handle_log_ack log(last 110) v1 -14402> 2013-04-03 10:39:38.646438 7fcdffcdb700 10 logged 2013-04-03 10:39:37.613775 mon.0 10.252.24.25:6789/0 110 : [INF] pgmap v29044: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1888B/s wr, 0op/s -14401> 2013-04-03 10:39:38.660210 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113489 ==== paxos(lease_ack lc 427374 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3537204054 0 0) 0x2787780 con 0x208a840 -14400> 2013-04-03 10:39:38.660247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos -14399> 2013-04-03 10:39:38.660264 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos version 394045 summary v 394045 -14398> 2013-04-03 10:39:38.660281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14397> 2013-04-03 10:39:39.586023 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 92 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140378 v187) v2 ==== 289+0+0 (3307655523 0 0) 0x2ce1000 con 0x2118420 -14396> 2013-04-03 10:39:39.586101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140378 v187) v2 from mds.? 10.252.58.31:6800/10908 -14395> 2013-04-03 10:39:39.586115 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140378 v187) v2 -- ?+0 0x29c5340 con 0x2118420 -14394> 2013-04-03 10:39:40.932256 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113490 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 93 v187) v2) to leader v1 ==== 560+0+0 (3365213601 0 0) 0x2a848c0 con 0x208a840 -14393> 2013-04-03 10:39:40.932277 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -14392> 2013-04-03 10:39:40.932288 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a832c0 from 10.252.24.88:6789/0 -14391> 2013-04-03 10:39:40.932340 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 93 v187) v2 from mds.0 10.252.24.25:6802/3144 -14390> 2013-04-03 10:39:40.932352 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 93 v187) v2 tid 17572) v2 -- ?+0 0x24f8b40 con 0x208a840 -14389> 2013-04-03 10:39:41.385306 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 92 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21749 v187) v2 ==== 289+0+0 (2340017216 0 0) 0x29c4dc0 con 0x200d420 -14388> 2013-04-03 10:39:41.385382 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21749 v187) v2 from mds.? 10.252.24.88:6800/17213 -14387> 2013-04-03 10:39:41.385396 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21749 v187) v2 -- ?+0 0x2a848c0 con 0x200d420 -14386> 2013-04-03 10:39:41.590051 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113491 ==== forward(pg_stats(1 pgs tid 32 v 605) v1) to leader v1 ==== 753+0+0 (3392732883 0 0) 0x2ce1000 con 0x208a840 -14385> 2013-04-03 10:39:41.590067 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -14384> 2013-04-03 10:39:41.590076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -14383> 2013-04-03 10:39:41.590111 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 preprocess_query pg_stats(1 pgs tid 32 v 605) v1 from osd.1 10.252.24.25:6805/3572 -14382> 2013-04-03 10:39:41.590136 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 prepare_update pg_stats(1 pgs tid 32 v 605) v1 from osd.1 10.252.24.25:6805/3572 -14381> 2013-04-03 10:39:41.590142 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 prepare_pg_stats pg_stats(1 pgs tid 32 v 605) v1 from osd.1 -14380> 2013-04-03 10:39:41.590149 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -14379> 2013-04-03 10:39:41.640333 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 encode_pending v 29045 -14378> 2013-04-03 10:39:41.640370 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427374) queue_proposal bl 650 bytes; ctx = 0x2c3d4e0 -14377> 2013-04-03 10:39:41.640380 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427374) propose_queued 427375 650 bytes -14376> 2013-04-03 10:39:41.641552 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427374 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4280 -14375> 2013-04-03 10:39:41.774285 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113492 ==== paxos(accept lc 427374 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4155200613 0 0) 0x2ab4280 con 0x208a840 -14374> 2013-04-03 10:39:41.774836 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427375 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787780 -14373> 2013-04-03 10:39:41.774861 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427375 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2ab4780 -14372> 2013-04-03 10:39:41.774936 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29044 update_from_paxos applying incremental 29045 -14371> 2013-04-03 10:39:41.774953 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 v29045: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3509B/s wr, 0op/s -14370> 2013-04-03 10:39:41.774972 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 send_pg_creates to 0 pgs -14369> 2013-04-03 10:39:41.774977 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 update_logger -14368> 2013-04-03 10:39:41.774987 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 create_pending v 29046 -14367> 2013-04-03 10:39:41.774996 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 _updated_stats for osd.1 10.252.24.25:6805/3572 -14366> 2013-04-03 10:39:41.775003 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 32) v1 tid 17573) v2 -- ?+0 0x24f8240 con 0x208a840 -14365> 2013-04-03 10:39:41.775021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 check_osd_map already seen 605 >= 605 -14364> 2013-04-03 10:39:41.775025 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 update_logger -14363> 2013-04-03 10:39:41.775036 7fcdffcdb700 0 log [INF] : pgmap v29045: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3509B/s wr, 0op/s -14362> 2013-04-03 10:39:41.775038 7fcdffcdb700 10 send_log to self -14361> 2013-04-03 10:39:41.775040 7fcdffcdb700 10 log_queue is 1 last_log 111 sent 110 num 1 unsent 1 sending 1 -14360> 2013-04-03 10:39:41.775041 7fcdffcdb700 10 will send 2013-04-03 10:39:41.775037 mon.0 10.252.24.25:6789/0 111 : [INF] pgmap v29045: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3509B/s wr, 0op/s -14359> 2013-04-03 10:39:41.775049 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24e8e00 -14358> 2013-04-03 10:39:41.775097 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos -14357> 2013-04-03 10:39:41.775116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos version 394045 summary v 394045 -14356> 2013-04-03 10:39:41.775135 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14355> 2013-04-03 10:39:41.775173 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24e8e00 con 0x1bac160 -14354> 2013-04-03 10:39:41.775197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos -14353> 2013-04-03 10:39:41.775213 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos version 394045 summary v 394045 -14352> 2013-04-03 10:39:41.775222 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14351> 2013-04-03 10:39:41.775233 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 preprocess_log log(1 entries) v1 from mon.0 -14350> 2013-04-03 10:39:41.775258 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14349> 2013-04-03 10:39:41.775271 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 prepare_log log(1 entries) v1 from mon.0 -14348> 2013-04-03 10:39:41.775282 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 logging 2013-04-03 10:39:41.775037 mon.0 10.252.24.25:6789/0 111 : [INF] pgmap v29045: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3509B/s wr, 0op/s -14347> 2013-04-03 10:39:41.786219 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113493 ==== paxos(lease_ack lc 427375 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2200021533 0 0) 0x2c6ec80 con 0x208a840 -14346> 2013-04-03 10:39:41.786256 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos -14345> 2013-04-03 10:39:41.786273 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 update_from_paxos version 394045 summary v 394045 -14344> 2013-04-03 10:39:41.786288 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14343> 2013-04-03 10:39:42.774680 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394045 encode_pending v394046 -14342> 2013-04-03 10:39:42.774709 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427375) queue_proposal bl 443 bytes; ctx = 0x2c3d0b0 -14341> 2013-04-03 10:39:42.774716 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427375) propose_queued 427376 443 bytes -14340> 2013-04-03 10:39:42.775775 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427375 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4c80 -14339> 2013-04-03 10:39:42.818925 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113494 ==== paxos(accept lc 427375 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2555646642 0 0) 0x2ab4c80 con 0x208a840 -14338> 2013-04-03 10:39:42.819216 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427376 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6ec80 -14337> 2013-04-03 10:39:42.819250 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427376 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2ab4280 -14336> 2013-04-03 10:39:42.819281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos -14335> 2013-04-03 10:39:42.819309 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos version 394046 summary v 394045 -14334> 2013-04-03 10:39:42.819323 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos latest full 393798 -14333> 2013-04-03 10:39:42.819342 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos applying incremental log 394046 2013-04-03 10:39:41.775037 mon.0 10.252.24.25:6789/0 111 : [INF] pgmap v29045: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3509B/s wr, 0op/s -14332> 2013-04-03 10:39:42.819390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 check_subs -14331> 2013-04-03 10:39:42.819421 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 create_pending v 394047 -14330> 2013-04-03 10:39:42.819444 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 _updated_log for mon.0 10.252.24.25:6789/0 -14329> 2013-04-03 10:39:42.819457 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 111) v1 -- ?+0 0x28a5540 con 0x1bac160 -14328> 2013-04-03 10:39:42.819509 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos -14327> 2013-04-03 10:39:42.819525 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos version 394046 summary v 394046 -14326> 2013-04-03 10:39:42.819885 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14325> 2013-04-03 10:39:42.819912 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 111) v1 ==== 0+0+0 (0 0 0) 0x28a5540 con 0x1bac160 -14324> 2013-04-03 10:39:42.819921 7fcdffcdb700 10 handle_log_ack log(last 111) v1 -14323> 2013-04-03 10:39:42.819922 7fcdffcdb700 10 logged 2013-04-03 10:39:41.775037 mon.0 10.252.24.25:6789/0 111 : [INF] pgmap v29045: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3509B/s wr, 0op/s -14322> 2013-04-03 10:39:42.830869 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113495 ==== paxos(lease_ack lc 427376 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (299195530 0 0) 0x2ab4280 con 0x208a840 -14321> 2013-04-03 10:39:42.830908 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos -14320> 2013-04-03 10:39:42.830925 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos version 394046 summary v 394046 -14319> 2013-04-03 10:39:42.830944 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14318> 2013-04-03 10:39:43.230092 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113496 ==== forward(pg_stats(1 pgs tid 11 v 605) v1) to leader v1 ==== 757+0+0 (3135121401 0 0) 0x2995dc0 con 0x208a840 -14317> 2013-04-03 10:39:43.230118 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -14316> 2013-04-03 10:39:43.230130 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -14315> 2013-04-03 10:39:43.230182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 preprocess_query pg_stats(1 pgs tid 11 v 605) v1 from osd.2 10.252.24.30:6801/5656 -14314> 2013-04-03 10:39:43.230206 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 prepare_update pg_stats(1 pgs tid 11 v 605) v1 from osd.2 10.252.24.30:6801/5656 -14313> 2013-04-03 10:39:43.230213 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 prepare_pg_stats pg_stats(1 pgs tid 11 v 605) v1 from osd.2 -14312> 2013-04-03 10:39:43.230220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -14311> 2013-04-03 10:39:43.348258 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 check_down_pgs -14310> 2013-04-03 10:39:43.348283 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 v29045: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 3509B/s wr, 0op/s -14309> 2013-04-03 10:39:43.348311 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -14308> 2013-04-03 10:39:43.348337 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -14307> 2013-04-03 10:39:43.348374 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -14306> 2013-04-03 10:39:43.348406 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos -14305> 2013-04-03 10:39:43.348425 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos version 394046 summary v 394046 -14304> 2013-04-03 10:39:43.348434 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 log -14303> 2013-04-03 10:39:43.348451 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14302> 2013-04-03 10:39:43.348470 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -14301> 2013-04-03 10:39:43.586140 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 93 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140379 v187) v2 ==== 289+0+0 (3863506462 0 0) 0x273a000 con 0x2118420 -14300> 2013-04-03 10:39:43.586186 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140379 v187) v2 from mds.? 10.252.58.31:6800/10908 -14299> 2013-04-03 10:39:43.586197 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140379 v187) v2 -- ?+0 0x2995dc0 con 0x2118420 -14298> 2013-04-03 10:39:43.819104 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 encode_pending v 29046 -14297> 2013-04-03 10:39:43.819143 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427376) queue_proposal bl 654 bytes; ctx = 0x2c3d310 -14296> 2013-04-03 10:39:43.819152 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427376) propose_queued 427377 654 bytes -14295> 2013-04-03 10:39:43.820523 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427376 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc280 -14294> 2013-04-03 10:39:43.847085 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113497 ==== paxos(accept lc 427376 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2844925999 0 0) 0x26dc280 con 0x208a840 -14293> 2013-04-03 10:39:43.847365 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427377 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4280 -14292> 2013-04-03 10:39:43.847397 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427377 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2ab4c80 -14291> 2013-04-03 10:39:43.847461 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29045 update_from_paxos applying incremental 29046 -14290> 2013-04-03 10:39:43.847480 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 v29046: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3936B/s wr, 0op/s -14289> 2013-04-03 10:39:43.847498 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 send_pg_creates to 0 pgs -14288> 2013-04-03 10:39:43.847503 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 update_logger -14287> 2013-04-03 10:39:43.847513 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 create_pending v 29047 -14286> 2013-04-03 10:39:43.847527 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 _updated_stats for osd.2 10.252.24.30:6801/5656 -14285> 2013-04-03 10:39:43.847535 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 11) v1 tid 17574) v2 -- ?+0 0x24f8b40 con 0x208a840 -14284> 2013-04-03 10:39:43.847545 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 check_osd_map already seen 605 >= 605 -14283> 2013-04-03 10:39:43.847548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 update_logger -14282> 2013-04-03 10:39:43.847560 7fcdffcdb700 0 log [INF] : pgmap v29046: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3936B/s wr, 0op/s -14281> 2013-04-03 10:39:43.847561 7fcdffcdb700 10 send_log to self -14280> 2013-04-03 10:39:43.847562 7fcdffcdb700 10 log_queue is 1 last_log 112 sent 111 num 1 unsent 1 sending 1 -14279> 2013-04-03 10:39:43.847564 7fcdffcdb700 10 will send 2013-04-03 10:39:43.847561 mon.0 10.252.24.25:6789/0 112 : [INF] pgmap v29046: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3936B/s wr, 0op/s -14278> 2013-04-03 10:39:43.847570 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24e8e00 -14277> 2013-04-03 10:39:43.847616 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos -14276> 2013-04-03 10:39:43.847634 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos version 394046 summary v 394046 -14275> 2013-04-03 10:39:43.847652 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14274> 2013-04-03 10:39:43.847673 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24e8e00 con 0x1bac160 -14273> 2013-04-03 10:39:43.847695 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos -14272> 2013-04-03 10:39:43.847710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos version 394046 summary v 394046 -14271> 2013-04-03 10:39:43.847718 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14270> 2013-04-03 10:39:43.847729 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 preprocess_log log(1 entries) v1 from mon.0 -14269> 2013-04-03 10:39:43.847753 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14268> 2013-04-03 10:39:43.847766 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 prepare_log log(1 entries) v1 from mon.0 -14267> 2013-04-03 10:39:43.847781 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 logging 2013-04-03 10:39:43.847561 mon.0 10.252.24.25:6789/0 112 : [INF] pgmap v29046: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3936B/s wr, 0op/s -14266> 2013-04-03 10:39:43.871960 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113498 ==== paxos(lease_ack lc 427377 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (358890663 0 0) 0x2ab4c80 con 0x208a840 -14265> 2013-04-03 10:39:43.872002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos -14264> 2013-04-03 10:39:43.872019 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 update_from_paxos version 394046 summary v 394046 -14263> 2013-04-03 10:39:43.872035 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14262> 2013-04-03 10:39:44.847287 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394046 encode_pending v394047 -14261> 2013-04-03 10:39:44.847316 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427377) queue_proposal bl 443 bytes; ctx = 0x2c3d2f0 -14260> 2013-04-03 10:39:44.847323 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427377) propose_queued 427378 443 bytes -14259> 2013-04-03 10:39:44.848555 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427377 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dcc80 -14258> 2013-04-03 10:39:44.871190 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113499 ==== paxos(accept lc 427377 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4156192441 0 0) 0x26dcc80 con 0x208a840 -14257> 2013-04-03 10:39:44.871506 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427378 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4c80 -14256> 2013-04-03 10:39:44.871539 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427378 fc 427326 pn 0 opn 0) v3 -- ?+0 0x26dc280 -14255> 2013-04-03 10:39:44.871575 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos -14254> 2013-04-03 10:39:44.871610 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394046 -14253> 2013-04-03 10:39:44.871625 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos latest full 393798 -14252> 2013-04-03 10:39:44.871645 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos applying incremental log 394047 2013-04-03 10:39:43.847561 mon.0 10.252.24.25:6789/0 112 : [INF] pgmap v29046: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3936B/s wr, 0op/s -14251> 2013-04-03 10:39:44.871692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 check_subs -14250> 2013-04-03 10:39:44.871724 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 create_pending v 394048 -14249> 2013-04-03 10:39:44.871747 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 _updated_log for mon.0 10.252.24.25:6789/0 -14248> 2013-04-03 10:39:44.871760 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 112) v1 -- ?+0 0x28a5e00 con 0x1bac160 -14247> 2013-04-03 10:39:44.871811 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos -14246> 2013-04-03 10:39:44.871827 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394047 -14245> 2013-04-03 10:39:44.871846 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14244> 2013-04-03 10:39:44.871871 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 112) v1 ==== 0+0+0 (0 0 0) 0x28a5e00 con 0x1bac160 -14243> 2013-04-03 10:39:44.871879 7fcdffcdb700 10 handle_log_ack log(last 112) v1 -14242> 2013-04-03 10:39:44.871880 7fcdffcdb700 10 logged 2013-04-03 10:39:43.847561 mon.0 10.252.24.25:6789/0 112 : [INF] pgmap v29046: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 3936B/s wr, 0op/s -14241> 2013-04-03 10:39:44.883153 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113500 ==== paxos(lease_ack lc 427378 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (130921654 0 0) 0x26dc280 con 0x208a840 -14240> 2013-04-03 10:39:44.883188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos -14239> 2013-04-03 10:39:44.883205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394047 -14238> 2013-04-03 10:39:44.883220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14237> 2013-04-03 10:39:44.932460 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113501 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 94 v187) v2) to leader v1 ==== 560+0+0 (3545200548 0 0) 0x2995840 con 0x208a840 -14236> 2013-04-03 10:39:44.932473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -14235> 2013-04-03 10:39:44.932482 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2996b80 from 10.252.24.88:6789/0 -14234> 2013-04-03 10:39:44.932512 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 94 v187) v2 from mds.0 10.252.24.25:6802/3144 -14233> 2013-04-03 10:39:44.932524 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 94 v187) v2 tid 17575) v2 -- ?+0 0x24f8240 con 0x208a840 -14232> 2013-04-03 10:39:45.385422 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 93 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21750 v187) v2 ==== 289+0+0 (4006221255 0 0) 0x29c5b80 con 0x200d420 -14231> 2013-04-03 10:39:45.385500 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21750 v187) v2 from mds.? 10.252.24.88:6800/17213 -14230> 2013-04-03 10:39:45.385514 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21750 v187) v2 -- ?+0 0x2995840 con 0x200d420 -14229> 2013-04-03 10:39:47.586286 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 94 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140380 v187) v2 ==== 289+0+0 (239161965 0 0) 0x273b600 con 0x2118420 -14228> 2013-04-03 10:39:47.586362 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140380 v187) v2 from mds.? 10.252.58.31:6800/10908 -14227> 2013-04-03 10:39:47.586375 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140380 v187) v2 -- ?+0 0x29c5b80 con 0x2118420 -14226> 2013-04-03 10:39:47.871607 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427378 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29e0000 -14225> 2013-04-03 10:39:47.872362 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113502 ==== paxos(lease_ack lc 427378 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1170109130 0 0) 0x29e0000 con 0x208a840 -14224> 2013-04-03 10:39:47.872413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos -14223> 2013-04-03 10:39:47.872433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394047 -14222> 2013-04-03 10:39:47.872451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14221> 2013-04-03 10:39:48.231001 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113503 ==== forward(pg_stats(1 pgs tid 12 v 605) v1) to leader v1 ==== 757+0+0 (3239578829 0 0) 0x273a000 con 0x208a840 -14220> 2013-04-03 10:39:48.231028 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -14219> 2013-04-03 10:39:48.231040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -14218> 2013-04-03 10:39:48.231092 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 preprocess_query pg_stats(1 pgs tid 12 v 605) v1 from osd.2 10.252.24.30:6801/5656 -14217> 2013-04-03 10:39:48.231117 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 prepare_update pg_stats(1 pgs tid 12 v 605) v1 from osd.2 10.252.24.30:6801/5656 -14216> 2013-04-03 10:39:48.231123 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 prepare_pg_stats pg_stats(1 pgs tid 12 v 605) v1 from osd.2 -14215> 2013-04-03 10:39:48.231130 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -14214> 2013-04-03 10:39:48.281313 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 encode_pending v 29047 -14213> 2013-04-03 10:39:48.281350 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427378) queue_proposal bl 654 bytes; ctx = 0x2c3d3e0 -14212> 2013-04-03 10:39:48.281359 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427378) propose_queued 427379 654 bytes -14211> 2013-04-03 10:39:48.282950 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427378 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0780 -14210> 2013-04-03 10:39:48.313376 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113504 ==== paxos(accept lc 427378 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4116310805 0 0) 0x29e0780 con 0x208a840 -14209> 2013-04-03 10:39:48.313764 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427379 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0000 -14208> 2013-04-03 10:39:48.313791 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427379 fc 427326 pn 0 opn 0) v3 -- ?+0 0x26dc280 -14207> 2013-04-03 10:39:48.313875 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29046 update_from_paxos applying incremental 29047 -14206> 2013-04-03 10:39:48.313897 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 v29047: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2467B/s wr, 0op/s -14205> 2013-04-03 10:39:48.313916 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 send_pg_creates to 0 pgs -14204> 2013-04-03 10:39:48.313919 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 update_logger -14203> 2013-04-03 10:39:48.313929 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 create_pending v 29048 -14202> 2013-04-03 10:39:48.313939 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 _updated_stats for osd.2 10.252.24.30:6801/5656 -14201> 2013-04-03 10:39:48.313951 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 12) v1 tid 17576) v2 -- ?+0 0x24f8b40 con 0x208a840 -14200> 2013-04-03 10:39:48.313962 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 check_osd_map already seen 605 >= 605 -14199> 2013-04-03 10:39:48.313965 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 update_logger -14198> 2013-04-03 10:39:48.313976 7fcdffcdb700 0 log [INF] : pgmap v29047: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2467B/s wr, 0op/s -14197> 2013-04-03 10:39:48.313977 7fcdffcdb700 10 send_log to self -14196> 2013-04-03 10:39:48.313978 7fcdffcdb700 10 log_queue is 1 last_log 113 sent 112 num 1 unsent 1 sending 1 -14195> 2013-04-03 10:39:48.313980 7fcdffcdb700 10 will send 2013-04-03 10:39:48.313977 mon.0 10.252.24.25:6789/0 113 : [INF] pgmap v29047: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2467B/s wr, 0op/s -14194> 2013-04-03 10:39:48.313986 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24e8e00 -14193> 2013-04-03 10:39:48.314038 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos -14192> 2013-04-03 10:39:48.314058 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394047 -14191> 2013-04-03 10:39:48.314076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14190> 2013-04-03 10:39:48.314098 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24e8e00 con 0x1bac160 -14189> 2013-04-03 10:39:48.314119 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos -14188> 2013-04-03 10:39:48.314134 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394047 -14187> 2013-04-03 10:39:48.314144 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14186> 2013-04-03 10:39:48.314155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 preprocess_log log(1 entries) v1 from mon.0 -14185> 2013-04-03 10:39:48.314178 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14184> 2013-04-03 10:39:48.314190 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 prepare_log log(1 entries) v1 from mon.0 -14183> 2013-04-03 10:39:48.314201 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 logging 2013-04-03 10:39:48.313977 mon.0 10.252.24.25:6789/0 113 : [INF] pgmap v29047: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2467B/s wr, 0op/s -14182> 2013-04-03 10:39:48.338565 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113505 ==== paxos(lease_ack lc 427379 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3421565021 0 0) 0x26dc280 con 0x208a840 -14181> 2013-04-03 10:39:48.338612 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos -14180> 2013-04-03 10:39:48.338630 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394047 -14179> 2013-04-03 10:39:48.338646 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14178> 2013-04-03 10:39:48.348582 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 check_down_pgs -14177> 2013-04-03 10:39:48.348611 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 v29047: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2467B/s wr, 0op/s -14176> 2013-04-03 10:39:48.348636 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -14175> 2013-04-03 10:39:48.348658 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -14174> 2013-04-03 10:39:48.348694 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -14173> 2013-04-03 10:39:48.348725 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos -14172> 2013-04-03 10:39:48.348742 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 update_from_paxos version 394047 summary v 394047 -14171> 2013-04-03 10:39:48.348751 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 log -14170> 2013-04-03 10:39:48.348767 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14169> 2013-04-03 10:39:48.348783 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -14168> 2013-04-03 10:39:48.932473 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113506 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 95 v187) v2) to leader v1 ==== 560+0+0 (1358791866 0 0) 0x29952c0 con 0x208a840 -14167> 2013-04-03 10:39:48.932489 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -14166> 2013-04-03 10:39:48.932498 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2995580 from 10.252.24.88:6789/0 -14165> 2013-04-03 10:39:48.932533 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 95 v187) v2 from mds.0 10.252.24.25:6802/3144 -14164> 2013-04-03 10:39:48.932543 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 95 v187) v2 tid 17577) v2 -- ?+0 0x24f8240 con 0x208a840 -14163> 2013-04-03 10:39:49.313583 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394047 encode_pending v394048 -14162> 2013-04-03 10:39:49.313615 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427379) queue_proposal bl 443 bytes; ctx = 0x2c3d4d0 -14161> 2013-04-03 10:39:49.313622 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427379) propose_queued 427380 443 bytes -14160> 2013-04-03 10:39:49.314704 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427379 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4da00 -14159> 2013-04-03 10:39:49.337904 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113507 ==== paxos(accept lc 427379 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1880036220 0 0) 0x2c4da00 con 0x208a840 -14158> 2013-04-03 10:39:49.338222 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427380 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dc280 -14157> 2013-04-03 10:39:49.338254 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427380 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29e0780 -14156> 2013-04-03 10:39:49.338296 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos -14155> 2013-04-03 10:39:49.338325 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos version 394048 summary v 394047 -14154> 2013-04-03 10:39:49.338478 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos latest full 393798 -14153> 2013-04-03 10:39:49.338502 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos applying incremental log 394048 2013-04-03 10:39:48.313977 mon.0 10.252.24.25:6789/0 113 : [INF] pgmap v29047: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2467B/s wr, 0op/s -14152> 2013-04-03 10:39:49.338547 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 check_subs -14151> 2013-04-03 10:39:49.338578 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 create_pending v 394049 -14150> 2013-04-03 10:39:49.338601 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 _updated_log for mon.0 10.252.24.25:6789/0 -14149> 2013-04-03 10:39:49.338618 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 113) v1 -- ?+0 0x28a58c0 con 0x1bac160 -14148> 2013-04-03 10:39:49.338671 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos -14147> 2013-04-03 10:39:49.338687 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos version 394048 summary v 394048 -14146> 2013-04-03 10:39:49.338706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14145> 2013-04-03 10:39:49.338729 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 113) v1 ==== 0+0+0 (0 0 0) 0x28a58c0 con 0x1bac160 -14144> 2013-04-03 10:39:49.338737 7fcdffcdb700 10 handle_log_ack log(last 113) v1 -14143> 2013-04-03 10:39:49.338740 7fcdffcdb700 10 logged 2013-04-03 10:39:48.313977 mon.0 10.252.24.25:6789/0 113 : [INF] pgmap v29047: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 2467B/s wr, 0op/s -14142> 2013-04-03 10:39:49.349897 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113508 ==== paxos(lease_ack lc 427380 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3723467627 0 0) 0x29e0780 con 0x208a840 -14141> 2013-04-03 10:39:49.349935 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos -14140> 2013-04-03 10:39:49.349951 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos version 394048 summary v 394048 -14139> 2013-04-03 10:39:49.349968 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14138> 2013-04-03 10:39:49.385483 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 94 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21751 v187) v2 ==== 289+0+0 (3450120890 0 0) 0x29c4840 con 0x200d420 -14137> 2013-04-03 10:39:49.385543 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21751 v187) v2 from mds.? 10.252.24.88:6800/17213 -14136> 2013-04-03 10:39:49.385556 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21751 v187) v2 -- ?+0 0x29952c0 con 0x200d420 -14135> 2013-04-03 10:39:51.586403 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 95 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140381 v187) v2 ==== 289+0+0 (758115600 0 0) 0x273b8c0 con 0x2118420 -14134> 2013-04-03 10:39:51.586481 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140381 v187) v2 from mds.? 10.252.58.31:6800/10908 -14133> 2013-04-03 10:39:51.586496 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140381 v187) v2 -- ?+0 0x29c4840 con 0x2118420 -14132> 2013-04-03 10:39:51.591143 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113509 ==== forward(pg_stats(1 pgs tid 33 v 605) v1) to leader v1 ==== 753+0+0 (3172638097 0 0) 0x273a000 con 0x208a840 -14131> 2013-04-03 10:39:51.591155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -14130> 2013-04-03 10:39:51.591166 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -14129> 2013-04-03 10:39:51.591199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 preprocess_query pg_stats(1 pgs tid 33 v 605) v1 from osd.1 10.252.24.25:6805/3572 -14128> 2013-04-03 10:39:51.591223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 prepare_update pg_stats(1 pgs tid 33 v 605) v1 from osd.1 10.252.24.25:6805/3572 -14127> 2013-04-03 10:39:51.591230 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 prepare_pg_stats pg_stats(1 pgs tid 33 v 605) v1 from osd.1 -14126> 2013-04-03 10:39:51.591236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -14125> 2013-04-03 10:39:51.641418 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 encode_pending v 29048 -14124> 2013-04-03 10:39:51.641460 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427380) queue_proposal bl 650 bytes; ctx = 0x2c3d4b0 -14123> 2013-04-03 10:39:51.641470 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427380) propose_queued 427381 650 bytes -14122> 2013-04-03 10:39:51.642403 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427380 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d000 -14121> 2013-04-03 10:39:51.806087 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113510 ==== paxos(accept lc 427380 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3351310984 0 0) 0x2c4d000 con 0x208a840 -14120> 2013-04-03 10:39:51.807059 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427381 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0780 -14119> 2013-04-03 10:39:51.807090 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427381 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c4da00 -14118> 2013-04-03 10:39:51.807156 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29047 update_from_paxos applying incremental 29048 -14117> 2013-04-03 10:39:51.807172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 v29048: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1309B/s wr, 0op/s -14116> 2013-04-03 10:39:51.807192 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 send_pg_creates to 0 pgs -14115> 2013-04-03 10:39:51.807197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 update_logger -14114> 2013-04-03 10:39:51.807207 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 create_pending v 29049 -14113> 2013-04-03 10:39:51.807217 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 _updated_stats for osd.1 10.252.24.25:6805/3572 -14112> 2013-04-03 10:39:51.807224 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 33) v1 tid 17578) v2 -- ?+0 0x24f8b40 con 0x208a840 -14111> 2013-04-03 10:39:51.807233 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 check_osd_map already seen 605 >= 605 -14110> 2013-04-03 10:39:51.807236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 update_logger -14109> 2013-04-03 10:39:51.807247 7fcdffcdb700 0 log [INF] : pgmap v29048: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1309B/s wr, 0op/s -14108> 2013-04-03 10:39:51.807249 7fcdffcdb700 10 send_log to self -14107> 2013-04-03 10:39:51.807255 7fcdffcdb700 10 log_queue is 1 last_log 114 sent 113 num 1 unsent 1 sending 1 -14106> 2013-04-03 10:39:51.807257 7fcdffcdb700 10 will send 2013-04-03 10:39:51.807248 mon.0 10.252.24.25:6789/0 114 : [INF] pgmap v29048: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1309B/s wr, 0op/s -14105> 2013-04-03 10:39:51.807263 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24e8e00 -14104> 2013-04-03 10:39:51.807308 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos -14103> 2013-04-03 10:39:51.807328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos version 394048 summary v 394048 -14102> 2013-04-03 10:39:51.807347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14101> 2013-04-03 10:39:51.807368 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24e8e00 con 0x1bac160 -14100> 2013-04-03 10:39:51.807390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos -14099> 2013-04-03 10:39:51.807405 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos version 394048 summary v 394048 -14098> 2013-04-03 10:39:51.807415 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14097> 2013-04-03 10:39:51.807427 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 preprocess_log log(1 entries) v1 from mon.0 -14096> 2013-04-03 10:39:51.807450 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14095> 2013-04-03 10:39:51.807463 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 prepare_log log(1 entries) v1 from mon.0 -14094> 2013-04-03 10:39:51.807473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 logging 2013-04-03 10:39:51.807248 mon.0 10.252.24.25:6789/0 114 : [INF] pgmap v29048: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1309B/s wr, 0op/s -14093> 2013-04-03 10:39:51.831262 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113511 ==== paxos(lease_ack lc 427381 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1938429561 0 0) 0x2c4da00 con 0x208a840 -14092> 2013-04-03 10:39:51.831302 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos -14091> 2013-04-03 10:39:51.831319 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 update_from_paxos version 394048 summary v 394048 -14090> 2013-04-03 10:39:51.831334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14089> 2013-04-03 10:39:51.831580 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113512 ==== forward(pg_stats(1 pgs tid 30447 v 605) v1) to leader v1 ==== 757+0+0 (617838038 0 0) 0x2995000 con 0x208a840 -14088> 2013-04-03 10:39:51.831592 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -14087> 2013-04-03 10:39:51.831599 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -14086> 2013-04-03 10:39:51.831625 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 preprocess_query pg_stats(1 pgs tid 30447 v 605) v1 from osd.3 10.252.24.88:6801/17538 -14085> 2013-04-03 10:39:51.831645 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 prepare_update pg_stats(1 pgs tid 30447 v 605) v1 from osd.3 10.252.24.88:6801/17538 -14084> 2013-04-03 10:39:51.831651 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 prepare_pg_stats pg_stats(1 pgs tid 30447 v 605) v1 from osd.3 -14083> 2013-04-03 10:39:51.831656 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1183 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -14082> 2013-04-03 10:39:52.806289 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394048 encode_pending v394049 -14081> 2013-04-03 10:39:52.806320 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427381) queue_proposal bl 443 bytes; ctx = 0x2c3d4c0 -14080> 2013-04-03 10:39:52.806330 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427381) propose_queued 427382 443 bytes -14079> 2013-04-03 10:39:52.808263 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427381 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4dc80 -14078> 2013-04-03 10:39:52.808363 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 encode_pending v 29049 -14077> 2013-04-03 10:39:52.808398 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427326..427381) queue_proposal bl 654 bytes; ctx = 0x2c3d4a0 -14076> 2013-04-03 10:39:52.808406 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427326..427381) propose_new_value not active; proposal queued -14075> 2013-04-03 10:39:52.838725 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113513 ==== paxos(accept lc 427381 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2920385822 0 0) 0x2c4dc80 con 0x208a840 -14074> 2013-04-03 10:39:52.839018 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427382 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4da00 -14073> 2013-04-03 10:39:52.839049 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427382 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c4d000 -14072> 2013-04-03 10:39:52.839088 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos -14071> 2013-04-03 10:39:52.839116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos version 394049 summary v 394048 -14070> 2013-04-03 10:39:52.839133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos latest full 393798 -14069> 2013-04-03 10:39:52.839152 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos applying incremental log 394049 2013-04-03 10:39:51.807248 mon.0 10.252.24.25:6789/0 114 : [INF] pgmap v29048: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1309B/s wr, 0op/s -14068> 2013-04-03 10:39:52.839202 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 check_subs -14067> 2013-04-03 10:39:52.839234 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 create_pending v 394050 -14066> 2013-04-03 10:39:52.839257 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 _updated_log for mon.0 10.252.24.25:6789/0 -14065> 2013-04-03 10:39:52.839272 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 114) v1 -- ?+0 0x2d98e00 con 0x1bac160 -14064> 2013-04-03 10:39:52.839297 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427382) propose_queued 427383 654 bytes -14063> 2013-04-03 10:39:52.839526 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427382 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dcc80 -14062> 2013-04-03 10:39:52.839571 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 114) v1 ==== 0+0+0 (0 0 0) 0x2d98e00 con 0x1bac160 -14061> 2013-04-03 10:39:52.839580 7fcdffcdb700 10 handle_log_ack log(last 114) v1 -14060> 2013-04-03 10:39:52.839581 7fcdffcdb700 10 logged 2013-04-03 10:39:51.807248 mon.0 10.252.24.25:6789/0 114 : [INF] pgmap v29048: 240 pgs: 240 active+clean; 254 MB data, 5655 MB used, 1067 GB / 1079 GB avail; 1309B/s wr, 0op/s -14059> 2013-04-03 10:39:52.850745 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113514 ==== paxos(lease_ack lc 427382 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3873479839 0 0) 0x26dcc80 con 0x208a840 -14058> 2013-04-03 10:39:52.863847 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113515 ==== paxos(accept lc 427382 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3081004573 0 0) 0x2c4d000 con 0x208a840 -14057> 2013-04-03 10:39:52.864130 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427383 fc 0 pn 17600 opn 0) v3 -- ?+0 0x26dcc80 -14056> 2013-04-03 10:39:52.864157 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427383 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c4dc80 -14055> 2013-04-03 10:39:52.864224 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29048 update_from_paxos applying incremental 29049 -14054> 2013-04-03 10:39:52.864243 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s -14053> 2013-04-03 10:39:52.864263 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 send_pg_creates to 0 pgs -14052> 2013-04-03 10:39:52.864268 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 update_logger -14051> 2013-04-03 10:39:52.864278 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 create_pending v 29050 -14050> 2013-04-03 10:39:52.864287 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 _updated_stats for osd.3 10.252.24.88:6801/17538 -14049> 2013-04-03 10:39:52.864294 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30447) v1 tid 17579) v2 -- ?+0 0x24f8240 con 0x208a840 -14048> 2013-04-03 10:39:52.864304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 check_osd_map already seen 605 >= 605 -14047> 2013-04-03 10:39:52.864307 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 update_logger -14046> 2013-04-03 10:39:52.864317 7fcdffcdb700 0 log [INF] : pgmap v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s -14045> 2013-04-03 10:39:52.864319 7fcdffcdb700 10 send_log to self -14044> 2013-04-03 10:39:52.864319 7fcdffcdb700 10 log_queue is 1 last_log 115 sent 114 num 1 unsent 1 sending 1 -14043> 2013-04-03 10:39:52.864326 7fcdffcdb700 10 will send 2013-04-03 10:39:52.864318 mon.0 10.252.24.25:6789/0 115 : [INF] pgmap v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s -14042> 2013-04-03 10:39:52.864334 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b76e00 -14041> 2013-04-03 10:39:52.864382 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos -14040> 2013-04-03 10:39:52.864399 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos version 394049 summary v 394049 -14039> 2013-04-03 10:39:52.864418 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14038> 2013-04-03 10:39:52.864439 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76e00 con 0x1bac160 -14037> 2013-04-03 10:39:52.864460 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos -14036> 2013-04-03 10:39:52.864476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos version 394049 summary v 394049 -14035> 2013-04-03 10:39:52.864485 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14034> 2013-04-03 10:39:52.864497 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 preprocess_log log(1 entries) v1 from mon.0 -14033> 2013-04-03 10:39:52.864517 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -14032> 2013-04-03 10:39:52.864530 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 prepare_log log(1 entries) v1 from mon.0 -14031> 2013-04-03 10:39:52.864540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 logging 2013-04-03 10:39:52.864318 mon.0 10.252.24.25:6789/0 115 : [INF] pgmap v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s -14030> 2013-04-03 10:39:52.889145 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113516 ==== paxos(lease_ack lc 427383 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3943937961 0 0) 0x2c4dc80 con 0x208a840 -14029> 2013-04-03 10:39:52.889187 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos -14028> 2013-04-03 10:39:52.889204 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos version 394049 summary v 394049 -14027> 2013-04-03 10:39:52.889221 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14026> 2013-04-03 10:39:52.932487 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113517 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 96 v187) v2) to leader v1 ==== 560+0+0 (296067521 0 0) 0x2ce2b80 con 0x208a840 -14025> 2013-04-03 10:39:52.932500 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -14024> 2013-04-03 10:39:52.932510 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ce2600 from 10.252.24.88:6789/0 -14023> 2013-04-03 10:39:52.932540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 96 v187) v2 from mds.0 10.252.24.25:6802/3144 -14022> 2013-04-03 10:39:52.932551 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 96 v187) v2 tid 17580) v2 -- ?+0 0x24f8b40 con 0x208a840 -14021> 2013-04-03 10:39:53.348900 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 check_down_pgs -14020> 2013-04-03 10:39:53.348931 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s -14019> 2013-04-03 10:39:53.348959 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -14018> 2013-04-03 10:39:53.348985 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -14017> 2013-04-03 10:39:53.349023 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -14016> 2013-04-03 10:39:53.349054 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos -14015> 2013-04-03 10:39:53.349074 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 update_from_paxos version 394049 summary v 394049 -14014> 2013-04-03 10:39:53.349083 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 log -14013> 2013-04-03 10:39:53.349100 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -14012> 2013-04-03 10:39:53.349120 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -14011> 2013-04-03 10:39:53.385649 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 95 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21752 v187) v2 ==== 289+0+0 (993270736 0 0) 0x29c5080 con 0x200d420 -14010> 2013-04-03 10:39:53.385724 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21752 v187) v2 from mds.? 10.252.24.88:6800/17213 -14009> 2013-04-03 10:39:53.385738 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21752 v187) v2 -- ?+0 0x2ce2b80 con 0x200d420 -14008> 2013-04-03 10:39:53.864026 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394049 encode_pending v394050 -14007> 2013-04-03 10:39:53.864053 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427383) queue_proposal bl 443 bytes; ctx = 0x2c3d0c0 -14006> 2013-04-03 10:39:53.864068 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427383) propose_queued 427384 443 bytes -14005> 2013-04-03 10:39:53.865517 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427383 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd280 -14004> 2013-04-03 10:39:53.978780 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113518 ==== paxos(accept lc 427383 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3604537608 0 0) 0x29fd280 con 0x208a840 -14003> 2013-04-03 10:39:53.979097 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427384 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4dc80 -14002> 2013-04-03 10:39:53.979132 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427384 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c4d000 -14001> 2013-04-03 10:39:53.979171 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos -14000> 2013-04-03 10:39:53.979198 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos version 394050 summary v 394049 -13999> 2013-04-03 10:39:53.979213 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos latest full 393798 -13998> 2013-04-03 10:39:53.979233 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos applying incremental log 394050 2013-04-03 10:39:52.864318 mon.0 10.252.24.25:6789/0 115 : [INF] pgmap v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s -13997> 2013-04-03 10:39:53.979278 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 check_subs -13996> 2013-04-03 10:39:53.979309 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 create_pending v 394051 -13995> 2013-04-03 10:39:53.979332 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 _updated_log for mon.0 10.252.24.25:6789/0 -13994> 2013-04-03 10:39:53.979345 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 115) v1 -- ?+0 0x2d98e00 con 0x1bac160 -13993> 2013-04-03 10:39:53.979395 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos -13992> 2013-04-03 10:39:53.979411 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos version 394050 summary v 394050 -13991> 2013-04-03 10:39:53.979433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13990> 2013-04-03 10:39:53.979455 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 115) v1 ==== 0+0+0 (0 0 0) 0x2d98e00 con 0x1bac160 -13989> 2013-04-03 10:39:53.979463 7fcdffcdb700 10 handle_log_ack log(last 115) v1 -13988> 2013-04-03 10:39:53.979466 7fcdffcdb700 10 logged 2013-04-03 10:39:52.864318 mon.0 10.252.24.25:6789/0 115 : [INF] pgmap v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s -13987> 2013-04-03 10:39:53.991987 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113519 ==== paxos(lease_ack lc 427384 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3168111888 0 0) 0x2c4d000 con 0x208a840 -13986> 2013-04-03 10:39:53.992022 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos -13985> 2013-04-03 10:39:53.992039 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos version 394050 summary v 394050 -13984> 2013-04-03 10:39:53.992057 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13983> 2013-04-03 10:39:55.586516 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 96 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140382 v187) v2 ==== 289+0+0 (1218194583 0 0) 0x273a2c0 con 0x2118420 -13982> 2013-04-03 10:39:55.586594 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140382 v187) v2 from mds.? 10.252.58.31:6800/10908 -13981> 2013-04-03 10:39:55.586608 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140382 v187) v2 -- ?+0 0x29c5080 con 0x2118420 -13980> 2013-04-03 10:39:56.932762 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113520 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 97 v187) v2) to leader v1 ==== 560+0+0 (418152386 0 0) 0x2995000 con 0x208a840 -13979> 2013-04-03 10:39:56.932782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -13978> 2013-04-03 10:39:56.932792 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ce28c0 from 10.252.24.88:6789/0 -13977> 2013-04-03 10:39:56.932844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 97 v187) v2 from mds.0 10.252.24.25:6802/3144 -13976> 2013-04-03 10:39:56.932856 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 97 v187) v2 tid 17581) v2 -- ?+0 0x24f8240 con 0x208a840 -13975> 2013-04-03 10:39:56.979202 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427384 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2c6e780 -13974> 2013-04-03 10:39:56.979940 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113521 ==== paxos(lease_ack lc 427384 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1073850983 0 0) 0x2c6e780 con 0x208a840 -13973> 2013-04-03 10:39:56.979986 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos -13972> 2013-04-03 10:39:56.980007 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos version 394050 summary v 394050 -13971> 2013-04-03 10:39:56.980028 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13970> 2013-04-03 10:39:57.385705 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 96 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21753 v187) v2 ==== 289+0+0 (408593581 0 0) 0x29c42c0 con 0x200d420 -13969> 2013-04-03 10:39:57.385775 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21753 v187) v2 from mds.? 10.252.24.88:6800/17213 -13968> 2013-04-03 10:39:57.385788 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21753 v187) v2 -- ?+0 0x2995000 con 0x200d420 -13967> 2013-04-03 10:39:58.349263 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29049 v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s -13966> 2013-04-03 10:39:58.349309 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -13965> 2013-04-03 10:39:58.349341 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -13964> 2013-04-03 10:39:58.349388 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -13963> 2013-04-03 10:39:58.349419 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos -13962> 2013-04-03 10:39:58.349439 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 update_from_paxos version 394050 summary v 394050 -13961> 2013-04-03 10:39:58.349576 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394050 log -13960> 2013-04-03 10:39:58.349600 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13959> 2013-04-03 10:39:58.349620 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -13958> 2013-04-03 10:39:58.485153 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) -13957> 2013-04-03 10:39:58.485198 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader -13956> 2013-04-03 10:39:59.586660 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 97 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140383 v187) v2 ==== 289+0+0 (1811059690 0 0) 0x273b340 con 0x2118420 -13955> 2013-04-03 10:39:59.586738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140383 v187) v2 from mds.? 10.252.58.31:6800/10908 -13954> 2013-04-03 10:39:59.586751 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140383 v187) v2 -- ?+0 0x29c42c0 con 0x2118420 -13953> 2013-04-03 10:39:59.979270 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427384 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29a1780 -13952> 2013-04-03 10:39:59.980164 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113522 ==== paxos(lease_ack lc 427384 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1273093141 0 0) 0x29a1780 con 0x208a840 -13951> 2013-04-03 10:39:59.980212 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos -13950> 2013-04-03 10:39:59.980231 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 -13949> 2013-04-03 10:39:59.980248 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13948> 2013-04-03 10:40:00.633185 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2acd280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33075/0 -13947> 2013-04-03 10:40:00.633218 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33075/0) -13946> 2013-04-03 10:40:00.633234 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd280 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -13945> 2013-04-03 10:40:00.633435 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd280 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -13944> 2013-04-03 10:40:00.932839 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113523 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 98 v187) v2) to leader v1 ==== 560+0+0 (53878215 0 0) 0x273a2c0 con 0x208a840 -13943> 2013-04-03 10:40:00.932853 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -13942> 2013-04-03 10:40:00.932862 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc5600 from 10.252.24.88:6789/0 -13941> 2013-04-03 10:40:00.932914 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 98 v187) v2 from mds.0 10.252.24.25:6802/3144 -13940> 2013-04-03 10:40:00.932926 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 98 v187) v2 tid 17582) v2 -- ?+0 0x24f8b40 con 0x208a840 -13939> 2013-04-03 10:40:01.385831 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 97 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21754 v187) v2 ==== 289+0+0 (2112447786 0 0) 0x29c5600 con 0x200d420 -13938> 2013-04-03 10:40:01.385905 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21754 v187) v2 from mds.? 10.252.24.88:6800/17213 -13937> 2013-04-03 10:40:01.385918 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21754 v187) v2 -- ?+0 0x273a2c0 con 0x200d420 -13936> 2013-04-03 10:40:02.979342 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427384 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29a1a00 -13935> 2013-04-03 10:40:02.980071 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113524 ==== paxos(lease_ack lc 427384 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3597080782 0 0) 0x29a1a00 con 0x208a840 -13934> 2013-04-03 10:40:02.980138 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos -13933> 2013-04-03 10:40:02.980158 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 -13932> 2013-04-03 10:40:02.980175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13931> 2013-04-03 10:40:03.349759 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29049 v29049: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 1583B/s wr, 0op/s -13930> 2013-04-03 10:40:03.349802 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -13929> 2013-04-03 10:40:03.349824 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -13928> 2013-04-03 10:40:03.349873 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -13927> 2013-04-03 10:40:03.349903 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos -13926> 2013-04-03 10:40:03.349922 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 -13925> 2013-04-03 10:40:03.349931 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394050 log -13924> 2013-04-03 10:40:03.349948 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13923> 2013-04-03 10:40:03.349966 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -13922> 2013-04-03 10:40:03.586764 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 98 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140384 v187) v2 ==== 289+0+0 (3661040854 0 0) 0x273a580 con 0x2118420 -13921> 2013-04-03 10:40:03.586838 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140384 v187) v2 from mds.? 10.252.58.31:6800/10908 -13920> 2013-04-03 10:40:03.586851 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140384 v187) v2 -- ?+0 0x29c5600 con 0x2118420 -13919> 2013-04-03 10:40:04.932953 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113525 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 99 v187) v2) to leader v1 ==== 560+0+0 (176145348 0 0) 0x273b340 con 0x208a840 -13918> 2013-04-03 10:40:04.932975 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -13917> 2013-04-03 10:40:04.932985 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4580 from 10.252.24.88:6789/0 -13916> 2013-04-03 10:40:04.933038 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 99 v187) v2 from mds.0 10.252.24.25:6802/3144 -13915> 2013-04-03 10:40:04.933048 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 99 v187) v2 tid 17583) v2 -- ?+0 0x24f8240 con 0x208a840 -13914> 2013-04-03 10:40:05.385908 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 98 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21755 v187) v2 ==== 289+0+0 (1585961559 0 0) 0x2145000 con 0x200d420 -13913> 2013-04-03 10:40:05.385982 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21755 v187) v2 from mds.? 10.252.24.88:6800/17213 -13912> 2013-04-03 10:40:05.385995 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21755 v187) v2 -- ?+0 0x273b340 con 0x200d420 -13911> 2013-04-03 10:40:05.979413 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427384 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2981a00 -13910> 2013-04-03 10:40:05.980190 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113526 ==== paxos(lease_ack lc 427384 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1888503652 0 0) 0x2981a00 con 0x208a840 -13909> 2013-04-03 10:40:05.980256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos -13908> 2013-04-03 10:40:05.980275 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 -13907> 2013-04-03 10:40:05.980293 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13906> 2013-04-03 10:40:07.586942 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 99 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140385 v187) v2 ==== 289+0+0 (4183405483 0 0) 0x273adc0 con 0x2118420 -13905> 2013-04-03 10:40:07.587016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140385 v187) v2 from mds.? 10.252.58.31:6800/10908 -13904> 2013-04-03 10:40:07.587029 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140385 v187) v2 -- ?+0 0x2145000 con 0x2118420 -13903> 2013-04-03 10:40:08.233433 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113527 ==== forward(pg_stats(1 pgs tid 13 v 605) v1) to leader v1 ==== 757+0+0 (610098801 0 0) 0x273a580 con 0x208a840 -13902> 2013-04-03 10:40:08.233458 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -13901> 2013-04-03 10:40:08.233468 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -13900> 2013-04-03 10:40:08.233518 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29049 preprocess_query pg_stats(1 pgs tid 13 v 605) v1 from osd.2 10.252.24.30:6801/5656 -13899> 2013-04-03 10:40:08.233542 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29049 prepare_update pg_stats(1 pgs tid 13 v 605) v1 from osd.2 10.252.24.30:6801/5656 -13898> 2013-04-03 10:40:08.233548 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29049 prepare_pg_stats pg_stats(1 pgs tid 13 v 605) v1 from osd.2 -13897> 2013-04-03 10:40:08.233555 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29049 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -13896> 2013-04-03 10:40:08.283750 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29049 encode_pending v 29050 -13895> 2013-04-03 10:40:08.283793 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427384) queue_proposal bl 654 bytes; ctx = 0x2c3d360 -13894> 2013-04-03 10:40:08.283801 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427384) propose_queued 427385 654 bytes -13893> 2013-04-03 10:40:08.285237 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427384 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2981000 -13892> 2013-04-03 10:40:08.319902 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113528 ==== paxos(accept lc 427384 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1156398108 0 0) 0x2981000 con 0x208a840 -13891> 2013-04-03 10:40:08.320866 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427385 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2981a00 -13890> 2013-04-03 10:40:08.320897 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427385 fc 427326 pn 0 opn 0) v3 -- ?+0 0x29a1a00 -13889> 2013-04-03 10:40:08.320973 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29049 update_from_paxos applying incremental 29050 -13888> 2013-04-03 10:40:08.320996 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 v29050: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 43870B/s wr, 0op/s -13887> 2013-04-03 10:40:08.321016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 send_pg_creates to 0 pgs -13886> 2013-04-03 10:40:08.321019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 update_logger -13885> 2013-04-03 10:40:08.321029 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 create_pending v 29051 -13884> 2013-04-03 10:40:08.321037 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29050 _updated_stats for osd.2 10.252.24.30:6801/5656 -13883> 2013-04-03 10:40:08.321043 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 13) v1 tid 17584) v2 -- ?+0 0x24f8b40 con 0x208a840 -13882> 2013-04-03 10:40:08.321054 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 check_osd_map already seen 605 >= 605 -13881> 2013-04-03 10:40:08.321057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 update_logger -13880> 2013-04-03 10:40:08.321067 7fcdffcdb700 0 log [INF] : pgmap v29050: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 43870B/s wr, 0op/s -13879> 2013-04-03 10:40:08.321069 7fcdffcdb700 10 send_log to self -13878> 2013-04-03 10:40:08.321070 7fcdffcdb700 10 log_queue is 1 last_log 116 sent 115 num 1 unsent 1 sending 1 -13877> 2013-04-03 10:40:08.321071 7fcdffcdb700 10 will send 2013-04-03 10:40:08.321068 mon.0 10.252.24.25:6789/0 116 : [INF] pgmap v29050: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 43870B/s wr, 0op/s -13876> 2013-04-03 10:40:08.321084 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b76e00 -13875> 2013-04-03 10:40:08.321136 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos -13874> 2013-04-03 10:40:08.321155 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 -13873> 2013-04-03 10:40:08.321173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13872> 2013-04-03 10:40:08.321194 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76e00 con 0x1bac160 -13871> 2013-04-03 10:40:08.321216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos -13870> 2013-04-03 10:40:08.321231 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 -13869> 2013-04-03 10:40:08.321240 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13868> 2013-04-03 10:40:08.321253 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 preprocess_log log(1 entries) v1 from mon.0 -13867> 2013-04-03 10:40:08.321278 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13866> 2013-04-03 10:40:08.321290 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 prepare_log log(1 entries) v1 from mon.0 -13865> 2013-04-03 10:40:08.321301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 logging 2013-04-03 10:40:08.321068 mon.0 10.252.24.25:6789/0 116 : [INF] pgmap v29050: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 43870B/s wr, 0op/s -13864> 2013-04-03 10:40:08.331932 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113529 ==== paxos(lease_ack lc 427385 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3651421357 0 0) 0x29a1a00 con 0x208a840 -13863> 2013-04-03 10:40:08.331974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos -13862> 2013-04-03 10:40:08.331990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 -13861> 2013-04-03 10:40:08.332006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13860> 2013-04-03 10:40:08.350082 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29050 check_down_pgs -13859> 2013-04-03 10:40:08.350118 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29050 v29050: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 43870B/s wr, 0op/s -13858> 2013-04-03 10:40:08.350141 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -13857> 2013-04-03 10:40:08.350165 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -13856> 2013-04-03 10:40:08.350201 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -13855> 2013-04-03 10:40:08.350231 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos -13854> 2013-04-03 10:40:08.350248 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394050 update_from_paxos version 394050 summary v 394050 -13853> 2013-04-03 10:40:08.350257 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394050 log -13852> 2013-04-03 10:40:08.350275 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13851> 2013-04-03 10:40:08.350291 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -13850> 2013-04-03 10:40:08.933230 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113530 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 100 v187) v2) to leader v1 ==== 560+0+0 (3306781801 0 0) 0x2cc4840 con 0x208a840 -13849> 2013-04-03 10:40:08.933251 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -13848> 2013-04-03 10:40:08.933261 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc5080 from 10.252.24.88:6789/0 -13847> 2013-04-03 10:40:08.933312 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 100 v187) v2 from mds.0 10.252.24.25:6802/3144 -13846> 2013-04-03 10:40:08.933323 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 100 v187) v2 tid 17585) v2 -- ?+0 0x24f8240 con 0x208a840 -13845> 2013-04-03 10:40:09.320107 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394050 encode_pending v394051 -13844> 2013-04-03 10:40:09.320136 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427385) queue_proposal bl 444 bytes; ctx = 0x2c3d370 -13843> 2013-04-03 10:40:09.320143 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427385) propose_queued 427386 444 bytes -13842> 2013-04-03 10:40:09.321239 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427385 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909a00 -13841> 2013-04-03 10:40:09.348022 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113531 ==== paxos(accept lc 427385 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3794997429 0 0) 0x2909a00 con 0x208a840 -13840> 2013-04-03 10:40:09.348315 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427386 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1a00 -13839> 2013-04-03 10:40:09.348350 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427386 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2981000 -13838> 2013-04-03 10:40:09.348383 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos -13837> 2013-04-03 10:40:09.348409 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos version 394051 summary v 394050 -13836> 2013-04-03 10:40:09.348423 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos latest full 393798 -13835> 2013-04-03 10:40:09.348444 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394051 update_from_paxos applying incremental log 394051 2013-04-03 10:40:08.321068 mon.0 10.252.24.25:6789/0 116 : [INF] pgmap v29050: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 43870B/s wr, 0op/s -13834> 2013-04-03 10:40:09.348489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 check_subs -13833> 2013-04-03 10:40:09.348520 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 create_pending v 394052 -13832> 2013-04-03 10:40:09.348543 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394051 _updated_log for mon.0 10.252.24.25:6789/0 -13831> 2013-04-03 10:40:09.348555 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 116) v1 -- ?+0 0x2d98c40 con 0x1bac160 -13830> 2013-04-03 10:40:09.348607 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos -13829> 2013-04-03 10:40:09.348623 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos version 394051 summary v 394051 -13828> 2013-04-03 10:40:09.348641 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13827> 2013-04-03 10:40:09.348663 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 116) v1 ==== 0+0+0 (0 0 0) 0x2d98c40 con 0x1bac160 -13826> 2013-04-03 10:40:09.348671 7fcdffcdb700 10 handle_log_ack log(last 116) v1 -13825> 2013-04-03 10:40:09.348672 7fcdffcdb700 10 logged 2013-04-03 10:40:08.321068 mon.0 10.252.24.25:6789/0 116 : [INF] pgmap v29050: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 43870B/s wr, 0op/s -13824> 2013-04-03 10:40:09.359820 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113532 ==== paxos(lease_ack lc 427386 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2592496039 0 0) 0x2981000 con 0x208a840 -13823> 2013-04-03 10:40:09.359856 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos -13822> 2013-04-03 10:40:09.359872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos version 394051 summary v 394051 -13821> 2013-04-03 10:40:09.359888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13820> 2013-04-03 10:40:09.386008 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 99 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21756 v187) v2 ==== 289+0+0 (3062818340 0 0) 0x2a848c0 con 0x200d420 -13819> 2013-04-03 10:40:09.386055 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21756 v187) v2 from mds.? 10.252.24.88:6800/17213 -13818> 2013-04-03 10:40:09.386066 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21756 v187) v2 -- ?+0 0x2cc4840 con 0x200d420 -13817> 2013-04-03 10:40:10.649180 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2acd000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33076/0 -13816> 2013-04-03 10:40:10.649208 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33076/0) -13815> 2013-04-03 10:40:10.649223 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd000 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -13814> 2013-04-03 10:40:10.649429 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2acd000 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -13813> 2013-04-03 10:40:11.587046 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 100 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140386 v187) v2 ==== 289+0+0 (2632610348 0 0) 0x2a51b00 con 0x2118420 -13812> 2013-04-03 10:40:11.587127 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140386 v187) v2 from mds.? 10.252.58.31:6800/10908 -13811> 2013-04-03 10:40:11.587140 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140386 v187) v2 -- ?+0 0x2a848c0 con 0x2118420 -13810> 2013-04-03 10:40:11.758958 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113533 ==== forward(pg_stats(1 pgs tid 30448 v 605) v1) to leader v1 ==== 757+0+0 (1051645684 0 0) 0x273a580 con 0x208a840 -13809> 2013-04-03 10:40:11.758972 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -13808> 2013-04-03 10:40:11.758981 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -13807> 2013-04-03 10:40:11.759014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 preprocess_query pg_stats(1 pgs tid 30448 v 605) v1 from osd.3 10.252.24.88:6801/17538 -13806> 2013-04-03 10:40:11.759038 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 prepare_update pg_stats(1 pgs tid 30448 v 605) v1 from osd.3 10.252.24.88:6801/17538 -13805> 2013-04-03 10:40:11.759043 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 prepare_pg_stats pg_stats(1 pgs tid 30448 v 605) v1 from osd.3 -13804> 2013-04-03 10:40:11.759049 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29050 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -13803> 2013-04-03 10:40:11.809230 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29050 encode_pending v 29051 -13802> 2013-04-03 10:40:11.809267 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427386) queue_proposal bl 654 bytes; ctx = 0x2c3d480 -13801> 2013-04-03 10:40:11.809276 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427386) propose_queued 427387 654 bytes -13800> 2013-04-03 10:40:11.810439 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427386 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909000 -13799> 2013-04-03 10:40:11.841565 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113534 ==== paxos(accept lc 427386 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2288935350 0 0) 0x2909000 con 0x208a840 -13798> 2013-04-03 10:40:11.841910 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427387 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2981000 -13797> 2013-04-03 10:40:11.841929 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427387 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2909a00 -13796> 2013-04-03 10:40:11.842000 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29050 update_from_paxos applying incremental 29051 -13795> 2013-04-03 10:40:11.842015 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 v29051: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 38371B/s wr, 0op/s -13794> 2013-04-03 10:40:11.842034 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 send_pg_creates to 0 pgs -13793> 2013-04-03 10:40:11.842041 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 update_logger -13792> 2013-04-03 10:40:11.842051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 create_pending v 29052 -13791> 2013-04-03 10:40:11.842060 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29051 _updated_stats for osd.3 10.252.24.88:6801/17538 -13790> 2013-04-03 10:40:11.842066 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30448) v1 tid 17586) v2 -- ?+0 0x24f8b40 con 0x208a840 -13789> 2013-04-03 10:40:11.842075 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 check_osd_map already seen 605 >= 605 -13788> 2013-04-03 10:40:11.842077 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 update_logger -13787> 2013-04-03 10:40:11.842088 7fcdffcdb700 0 log [INF] : pgmap v29051: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 38371B/s wr, 0op/s -13786> 2013-04-03 10:40:11.842089 7fcdffcdb700 10 send_log to self -13785> 2013-04-03 10:40:11.842090 7fcdffcdb700 10 log_queue is 1 last_log 117 sent 116 num 1 unsent 1 sending 1 -13784> 2013-04-03 10:40:11.842092 7fcdffcdb700 10 will send 2013-04-03 10:40:11.842089 mon.0 10.252.24.25:6789/0 117 : [INF] pgmap v29051: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 38371B/s wr, 0op/s -13783> 2013-04-03 10:40:11.842100 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b76e00 -13782> 2013-04-03 10:40:11.842143 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos -13781> 2013-04-03 10:40:11.842162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos version 394051 summary v 394051 -13780> 2013-04-03 10:40:11.842180 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13779> 2013-04-03 10:40:11.842201 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76e00 con 0x1bac160 -13778> 2013-04-03 10:40:11.842222 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos -13777> 2013-04-03 10:40:11.842237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos version 394051 summary v 394051 -13776> 2013-04-03 10:40:11.842246 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13775> 2013-04-03 10:40:11.842258 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 preprocess_log log(1 entries) v1 from mon.0 -13774> 2013-04-03 10:40:11.842285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13773> 2013-04-03 10:40:11.842297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 prepare_log log(1 entries) v1 from mon.0 -13772> 2013-04-03 10:40:11.842308 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 logging 2013-04-03 10:40:11.842089 mon.0 10.252.24.25:6789/0 117 : [INF] pgmap v29051: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 38371B/s wr, 0op/s -13771> 2013-04-03 10:40:11.853441 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113535 ==== paxos(lease_ack lc 427387 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1573783518 0 0) 0x2909a00 con 0x208a840 -13770> 2013-04-03 10:40:11.853481 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos -13769> 2013-04-03 10:40:11.853498 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394051 update_from_paxos version 394051 summary v 394051 -13768> 2013-04-03 10:40:11.853514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13767> 2013-04-03 10:40:12.841761 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394051 encode_pending v394052 -13766> 2013-04-03 10:40:12.841788 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427387) queue_proposal bl 444 bytes; ctx = 0x2c3d2e0 -13765> 2013-04-03 10:40:12.841794 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427387) propose_queued 427388 444 bytes -13764> 2013-04-03 10:40:12.842841 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427387 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909780 -13763> 2013-04-03 10:40:12.871097 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113536 ==== paxos(accept lc 427387 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1820844723 0 0) 0x2909780 con 0x208a840 -13762> 2013-04-03 10:40:12.871948 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427388 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909a00 -13761> 2013-04-03 10:40:12.871983 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427388 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2909000 -13760> 2013-04-03 10:40:12.872019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos -13759> 2013-04-03 10:40:12.872044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394051 -13758> 2013-04-03 10:40:12.872062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos latest full 393798 -13757> 2013-04-03 10:40:12.872081 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394052 update_from_paxos applying incremental log 394052 2013-04-03 10:40:11.842089 mon.0 10.252.24.25:6789/0 117 : [INF] pgmap v29051: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 38371B/s wr, 0op/s -13756> 2013-04-03 10:40:12.872125 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 check_subs -13755> 2013-04-03 10:40:12.872156 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 create_pending v 394053 -13754> 2013-04-03 10:40:12.872178 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394052 _updated_log for mon.0 10.252.24.25:6789/0 -13753> 2013-04-03 10:40:12.872191 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 117) v1 -- ?+0 0x2d98a80 con 0x1bac160 -13752> 2013-04-03 10:40:12.872242 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos -13751> 2013-04-03 10:40:12.872260 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 -13750> 2013-04-03 10:40:12.872279 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13749> 2013-04-03 10:40:12.872300 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 117) v1 ==== 0+0+0 (0 0 0) 0x2d98a80 con 0x1bac160 -13748> 2013-04-03 10:40:12.872308 7fcdffcdb700 10 handle_log_ack log(last 117) v1 -13747> 2013-04-03 10:40:12.872310 7fcdffcdb700 10 logged 2013-04-03 10:40:11.842089 mon.0 10.252.24.25:6789/0 117 : [INF] pgmap v29051: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 38371B/s wr, 0op/s -13746> 2013-04-03 10:40:12.896318 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113537 ==== paxos(lease_ack lc 427388 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3318641652 0 0) 0x2909000 con 0x208a840 -13745> 2013-04-03 10:40:12.896362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos -13744> 2013-04-03 10:40:12.896378 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 -13743> 2013-04-03 10:40:12.896394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13742> 2013-04-03 10:40:12.933168 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113538 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 101 v187) v2) to leader v1 ==== 560+0+0 (409762108 0 0) 0x29c4840 con 0x208a840 -13741> 2013-04-03 10:40:12.933181 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -13740> 2013-04-03 10:40:12.933189 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c5b80 from 10.252.24.88:6789/0 -13739> 2013-04-03 10:40:12.933224 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 101 v187) v2 from mds.0 10.252.24.25:6802/3144 -13738> 2013-04-03 10:40:12.933235 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 101 v187) v2 tid 17587) v2 -- ?+0 0x24f8240 con 0x208a840 -13737> 2013-04-03 10:40:13.350433 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29051 check_down_pgs -13736> 2013-04-03 10:40:13.350468 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29051 v29051: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 38371B/s wr, 0op/s -13735> 2013-04-03 10:40:13.350496 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -13734> 2013-04-03 10:40:13.350520 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -13733> 2013-04-03 10:40:13.350556 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -13732> 2013-04-03 10:40:13.350592 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos -13731> 2013-04-03 10:40:13.350611 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 -13730> 2013-04-03 10:40:13.350620 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394052 log -13729> 2013-04-03 10:40:13.350637 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13728> 2013-04-03 10:40:13.350656 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -13727> 2013-04-03 10:40:13.386134 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 100 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21757 v187) v2 ==== 289+0+0 (2514508121 0 0) 0x2a83840 con 0x200d420 -13726> 2013-04-03 10:40:13.386184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21757 v187) v2 from mds.? 10.252.24.88:6800/17213 -13725> 2013-04-03 10:40:13.386195 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21757 v187) v2 -- ?+0 0x29c4840 con 0x200d420 -13724> 2013-04-03 10:40:15.587110 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 101 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140387 v187) v2 ==== 289+0+0 (3213151569 0 0) 0x2a51000 con 0x2118420 -13723> 2013-04-03 10:40:15.587184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140387 v187) v2 from mds.? 10.252.58.31:6800/10908 -13722> 2013-04-03 10:40:15.587201 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140387 v187) v2 -- ?+0 0x2a83840 con 0x2118420 -13721> 2013-04-03 10:40:15.872066 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427388 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2ab4780 -13720> 2013-04-03 10:40:15.872857 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113539 ==== paxos(lease_ack lc 427388 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (1870170037 0 0) 0x2ab4780 con 0x208a840 -13719> 2013-04-03 10:40:15.872929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos -13718> 2013-04-03 10:40:15.872949 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 -13717> 2013-04-03 10:40:15.872966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13716> 2013-04-03 10:40:16.937385 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113540 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 102 v187) v2) to leader v1 ==== 560+0+0 (906709248 0 0) 0x273a580 con 0x208a840 -13715> 2013-04-03 10:40:16.937406 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -13714> 2013-04-03 10:40:16.937415 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c5340 from 10.252.24.88:6789/0 -13713> 2013-04-03 10:40:16.937470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 102 v187) v2 from mds.0 10.252.24.25:6802/3144 -13712> 2013-04-03 10:40:16.937482 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 102 v187) v2 tid 17588) v2 -- ?+0 0x24f8b40 con 0x208a840 -13711> 2013-04-03 10:40:17.386247 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 101 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21758 v187) v2 ==== 289+0+0 (4032010462 0 0) 0x2a84340 con 0x200d420 -13710> 2013-04-03 10:40:17.386317 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21758 v187) v2 from mds.? 10.252.24.88:6800/17213 -13709> 2013-04-03 10:40:17.386329 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21758 v187) v2 -- ?+0 0x273a580 con 0x200d420 -13708> 2013-04-03 10:40:18.350773 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29051 v29051: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 38371B/s wr, 0op/s -13707> 2013-04-03 10:40:18.350814 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -13706> 2013-04-03 10:40:18.350837 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -13705> 2013-04-03 10:40:18.350879 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -13704> 2013-04-03 10:40:18.350910 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos -13703> 2013-04-03 10:40:18.350929 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 -13702> 2013-04-03 10:40:18.350938 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394052 log -13701> 2013-04-03 10:40:18.350955 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13700> 2013-04-03 10:40:18.350974 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -13699> 2013-04-03 10:40:18.872133 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427388 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2ab4000 -13698> 2013-04-03 10:40:18.872906 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113541 ==== paxos(lease_ack lc 427388 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2485379187 0 0) 0x2ab4000 con 0x208a840 -13697> 2013-04-03 10:40:18.872973 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos -13696> 2013-04-03 10:40:18.872993 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 -13695> 2013-04-03 10:40:18.873010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13694> 2013-04-03 10:40:19.587259 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 102 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140388 v187) v2 ==== 289+0+0 (1468920098 0 0) 0x2a528c0 con 0x2118420 -13693> 2013-04-03 10:40:19.587332 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140388 v187) v2 from mds.? 10.252.58.31:6800/10908 -13692> 2013-04-03 10:40:19.587347 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140388 v187) v2 -- ?+0 0x2a84340 con 0x2118420 -13691> 2013-04-03 10:40:20.933466 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113542 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 103 v187) v2) to leader v1 ==== 560+0+0 (1061328643 0 0) 0x2a51000 con 0x208a840 -13690> 2013-04-03 10:40:20.933486 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -13689> 2013-04-03 10:40:20.933496 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x29c58c0 from 10.252.24.88:6789/0 -13688> 2013-04-03 10:40:20.933549 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 103 v187) v2 from mds.0 10.252.24.25:6802/3144 -13687> 2013-04-03 10:40:20.933560 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 103 v187) v2 tid 17589) v2 -- ?+0 0x24f8240 con 0x208a840 -13686> 2013-04-03 10:40:21.386390 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 102 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21759 v187) v2 ==== 289+0+0 (3544000419 0 0) 0x2a83b00 con 0x200d420 -13685> 2013-04-03 10:40:21.386467 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21759 v187) v2 from mds.? 10.252.24.88:6800/17213 -13684> 2013-04-03 10:40:21.386480 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21759 v187) v2 -- ?+0 0x2996340 con 0x200d420 -13683> 2013-04-03 10:40:21.594478 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113543 ==== forward(pg_stats(1 pgs tid 34 v 605) v1) to leader v1 ==== 753+0+0 (2646551325 0 0) 0x2a528c0 con 0x208a840 -13682> 2013-04-03 10:40:21.594493 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -13681> 2013-04-03 10:40:21.594502 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -13680> 2013-04-03 10:40:21.594537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 preprocess_query pg_stats(1 pgs tid 34 v 605) v1 from osd.1 10.252.24.25:6805/3572 -13679> 2013-04-03 10:40:21.594561 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 prepare_update pg_stats(1 pgs tid 34 v 605) v1 from osd.1 10.252.24.25:6805/3572 -13678> 2013-04-03 10:40:21.594567 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 prepare_pg_stats pg_stats(1 pgs tid 34 v 605) v1 from osd.1 -13677> 2013-04-03 10:40:21.594573 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29051 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -13676> 2013-04-03 10:40:21.644758 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29051 encode_pending v 29052 -13675> 2013-04-03 10:40:21.644789 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427388) queue_proposal bl 650 bytes; ctx = 0x2c3d5b0 -13674> 2013-04-03 10:40:21.644797 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427388) propose_queued 427389 650 bytes -13673> 2013-04-03 10:40:21.645749 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427388 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4500 -13672> 2013-04-03 10:40:21.767081 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113544 ==== paxos(accept lc 427388 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4144110431 0 0) 0x2ab4500 con 0x208a840 -13671> 2013-04-03 10:40:21.767443 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427389 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4000 -13670> 2013-04-03 10:40:21.767465 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427389 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2ab4780 -13669> 2013-04-03 10:40:21.767558 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29051 update_from_paxos applying incremental 29052 -13668> 2013-04-03 10:40:21.767577 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 v29052: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 306B/s wr, 0op/s -13667> 2013-04-03 10:40:21.767596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 send_pg_creates to 0 pgs -13666> 2013-04-03 10:40:21.767598 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 update_logger -13665> 2013-04-03 10:40:21.767608 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 create_pending v 29053 -13664> 2013-04-03 10:40:21.767616 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29052 _updated_stats for osd.1 10.252.24.25:6805/3572 -13663> 2013-04-03 10:40:21.767624 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 34) v1 tid 17590) v2 -- ?+0 0x24f8b40 con 0x208a840 -13662> 2013-04-03 10:40:21.767634 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 check_osd_map already seen 605 >= 605 -13661> 2013-04-03 10:40:21.767637 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 update_logger -13660> 2013-04-03 10:40:21.767647 7fcdffcdb700 0 log [INF] : pgmap v29052: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 306B/s wr, 0op/s -13659> 2013-04-03 10:40:21.767649 7fcdffcdb700 10 send_log to self -13658> 2013-04-03 10:40:21.767649 7fcdffcdb700 10 log_queue is 1 last_log 118 sent 117 num 1 unsent 1 sending 1 -13657> 2013-04-03 10:40:21.767651 7fcdffcdb700 10 will send 2013-04-03 10:40:21.767648 mon.0 10.252.24.25:6789/0 118 : [INF] pgmap v29052: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 306B/s wr, 0op/s -13656> 2013-04-03 10:40:21.767660 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b76e00 -13655> 2013-04-03 10:40:21.767710 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos -13654> 2013-04-03 10:40:21.767729 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 -13653> 2013-04-03 10:40:21.767748 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13652> 2013-04-03 10:40:21.767769 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76e00 con 0x1bac160 -13651> 2013-04-03 10:40:21.767790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos -13650> 2013-04-03 10:40:21.767808 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 -13649> 2013-04-03 10:40:21.767817 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13648> 2013-04-03 10:40:21.767828 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 preprocess_log log(1 entries) v1 from mon.0 -13647> 2013-04-03 10:40:21.767852 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13646> 2013-04-03 10:40:21.767864 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 prepare_log log(1 entries) v1 from mon.0 -13645> 2013-04-03 10:40:21.767874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 logging 2013-04-03 10:40:21.767648 mon.0 10.252.24.25:6789/0 118 : [INF] pgmap v29052: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 306B/s wr, 0op/s -13644> 2013-04-03 10:40:21.780310 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113545 ==== paxos(lease_ack lc 427389 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (13254711 0 0) 0x2ab4780 con 0x208a840 -13643> 2013-04-03 10:40:21.780347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos -13642> 2013-04-03 10:40:21.780363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394052 update_from_paxos version 394052 summary v 394052 -13641> 2013-04-03 10:40:21.780377 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13640> 2013-04-03 10:40:21.780641 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113546 ==== forward(pg_stats(1 pgs tid 30449 v 605) v1) to leader v1 ==== 757+0+0 (1565332037 0 0) 0x29c4580 con 0x208a840 -13639> 2013-04-03 10:40:21.780652 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -13638> 2013-04-03 10:40:21.780660 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -13637> 2013-04-03 10:40:21.780687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 preprocess_query pg_stats(1 pgs tid 30449 v 605) v1 from osd.3 10.252.24.88:6801/17538 -13636> 2013-04-03 10:40:21.780707 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 prepare_update pg_stats(1 pgs tid 30449 v 605) v1 from osd.3 10.252.24.88:6801/17538 -13635> 2013-04-03 10:40:21.780713 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 prepare_pg_stats pg_stats(1 pgs tid 30449 v 605) v1 from osd.3 -13634> 2013-04-03 10:40:21.780718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29052 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -13633> 2013-04-03 10:40:22.767291 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29052 encode_pending v 29053 -13632> 2013-04-03 10:40:22.767330 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427389) queue_proposal bl 654 bytes; ctx = 0x2c3d390 -13631> 2013-04-03 10:40:22.767339 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427389) propose_queued 427390 654 bytes -13630> 2013-04-03 10:40:22.769843 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427389 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4a00 -13629> 2013-04-03 10:40:22.769947 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394052 encode_pending v394053 -13628> 2013-04-03 10:40:22.769974 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427326..427389) queue_proposal bl 442 bytes; ctx = 0x2c3d470 -13627> 2013-04-03 10:40:22.769980 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427326..427389) propose_new_value not active; proposal queued -13626> 2013-04-03 10:40:22.794916 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113547 ==== paxos(accept lc 427389 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (466216807 0 0) 0x2ab4a00 con 0x208a840 -13625> 2013-04-03 10:40:22.795227 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427390 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4780 -13624> 2013-04-03 10:40:22.795263 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427390 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2ab4500 -13623> 2013-04-03 10:40:22.795342 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29052 update_from_paxos applying incremental 29053 -13622> 2013-04-03 10:40:22.795361 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 v29053: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 467B/s wr, 0op/s -13621> 2013-04-03 10:40:22.795380 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 send_pg_creates to 0 pgs -13620> 2013-04-03 10:40:22.795383 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 update_logger -13619> 2013-04-03 10:40:22.795392 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 create_pending v 29054 -13618> 2013-04-03 10:40:22.795400 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29053 _updated_stats for osd.3 10.252.24.88:6801/17538 -13617> 2013-04-03 10:40:22.795407 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30449) v1 tid 17591) v2 -- ?+0 0x24f8240 con 0x208a840 -13616> 2013-04-03 10:40:22.795418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 check_osd_map already seen 605 >= 605 -13615> 2013-04-03 10:40:22.795421 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 update_logger -13614> 2013-04-03 10:40:22.795430 7fcdffcdb700 0 log [INF] : pgmap v29053: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 467B/s wr, 0op/s -13613> 2013-04-03 10:40:22.795432 7fcdffcdb700 10 send_log to self -13612> 2013-04-03 10:40:22.795433 7fcdffcdb700 10 log_queue is 2 last_log 119 sent 118 num 2 unsent 1 sending 1 -13611> 2013-04-03 10:40:22.795434 7fcdffcdb700 10 will send 2013-04-03 10:40:21.767648 mon.0 10.252.24.25:6789/0 118 : [INF] pgmap v29052: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 306B/s wr, 0op/s -13610> 2013-04-03 10:40:22.795440 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b76600 -13609> 2013-04-03 10:40:22.795468 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427390) propose_queued 427391 442 bytes -13608> 2013-04-03 10:40:22.795678 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427390 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909000 -13607> 2013-04-03 10:40:22.795710 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76600 con 0x1bac160 -13606> 2013-04-03 10:40:22.820078 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113548 ==== paxos(lease_ack lc 427390 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (3481676456 0 0) 0x2909000 con 0x208a840 -13605> 2013-04-03 10:40:22.833091 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113549 ==== paxos(accept lc 427390 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3958004879 0 0) 0x2ab4500 con 0x208a840 -13604> 2013-04-03 10:40:22.834154 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427391 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909000 -13603> 2013-04-03 10:40:22.834172 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427391 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2ab4a00 -13602> 2013-04-03 10:40:22.834209 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos -13601> 2013-04-03 10:40:22.834233 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394052 -13600> 2013-04-03 10:40:22.834249 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos latest full 393798 -13599> 2013-04-03 10:40:22.834266 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394053 update_from_paxos applying incremental log 394053 2013-04-03 10:40:21.767648 mon.0 10.252.24.25:6789/0 118 : [INF] pgmap v29052: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 306B/s wr, 0op/s -13598> 2013-04-03 10:40:22.834309 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 check_subs -13597> 2013-04-03 10:40:22.834340 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 create_pending v 394054 -13596> 2013-04-03 10:40:22.834362 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394053 _updated_log for mon.0 10.252.24.25:6789/0 -13595> 2013-04-03 10:40:22.834374 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 118) v1 -- ?+0 0x2d98700 con 0x1bac160 -13594> 2013-04-03 10:40:22.834394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos -13593> 2013-04-03 10:40:22.834408 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 -13592> 2013-04-03 10:40:22.834418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13591> 2013-04-03 10:40:22.834430 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 preprocess_log log(1 entries) v1 from mon.0 -13590> 2013-04-03 10:40:22.834447 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 nothing new -13589> 2013-04-03 10:40:22.834494 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos -13588> 2013-04-03 10:40:22.834509 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 -13587> 2013-04-03 10:40:22.834527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13586> 2013-04-03 10:40:22.834549 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 118) v1 ==== 0+0+0 (0 0 0) 0x2d98700 con 0x1bac160 -13585> 2013-04-03 10:40:22.834557 7fcdffcdb700 10 handle_log_ack log(last 118) v1 -13584> 2013-04-03 10:40:22.834558 7fcdffcdb700 10 logged 2013-04-03 10:40:21.767648 mon.0 10.252.24.25:6789/0 118 : [INF] pgmap v29052: 240 pgs: 240 active+clean; 254 MB data, 5656 MB used, 1067 GB / 1079 GB avail; 306B/s wr, 0op/s -13583> 2013-04-03 10:40:22.845133 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113550 ==== paxos(lease_ack lc 427391 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (158465504 0 0) 0x2ab4a00 con 0x208a840 -13582> 2013-04-03 10:40:22.845178 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos -13581> 2013-04-03 10:40:22.845195 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 -13580> 2013-04-03 10:40:22.845211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13579> 2013-04-03 10:40:23.351120 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29053 check_down_pgs -13578> 2013-04-03 10:40:23.351150 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29053 v29053: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 467B/s wr, 0op/s -13577> 2013-04-03 10:40:23.351178 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -13576> 2013-04-03 10:40:23.351203 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -13575> 2013-04-03 10:40:23.351246 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -13574> 2013-04-03 10:40:23.351277 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos -13573> 2013-04-03 10:40:23.351296 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 -13572> 2013-04-03 10:40:23.351305 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394053 log -13571> 2013-04-03 10:40:23.351322 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13570> 2013-04-03 10:40:23.351341 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -13569> 2013-04-03 10:40:23.587366 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 103 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140389 v187) v2 ==== 289+0+0 (1961053791 0 0) 0x2a51580 con 0x2118420 -13568> 2013-04-03 10:40:23.587413 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140389 v187) v2 from mds.? 10.252.58.31:6800/10908 -13567> 2013-04-03 10:40:23.587424 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140389 v187) v2 -- ?+0 0x29c4580 con 0x2118420 -13566> 2013-04-03 10:40:24.933638 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113551 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 104 v187) v2) to leader v1 ==== 560+0+0 (3554275190 0 0) 0x29c4000 con 0x208a840 -13565> 2013-04-03 10:40:24.933658 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -13564> 2013-04-03 10:40:24.933667 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2995dc0 from 10.252.24.88:6789/0 -13563> 2013-04-03 10:40:24.933719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 104 v187) v2 from mds.0 10.252.24.25:6802/3144 -13562> 2013-04-03 10:40:24.933730 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 104 v187) v2 tid 17592) v2 -- ?+0 0x24f8b40 con 0x208a840 -13561> 2013-04-03 10:40:25.386509 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 103 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21760 v187) v2 ==== 289+0+0 (2806337285 0 0) 0x20e4340 con 0x200d420 -13560> 2013-04-03 10:40:25.386584 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21760 v187) v2 from mds.? 10.252.24.88:6800/17213 -13559> 2013-04-03 10:40:25.386597 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21760 v187) v2 -- ?+0 0x29c4000 con 0x200d420 -13558> 2013-04-03 10:40:25.834249 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427391 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2787780 -13557> 2013-04-03 10:40:25.835123 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113552 ==== paxos(lease_ack lc 427391 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (252115238 0 0) 0x2787780 con 0x208a840 -13556> 2013-04-03 10:40:25.835190 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos -13555> 2013-04-03 10:40:25.835210 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 -13554> 2013-04-03 10:40:25.835227 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13553> 2013-04-03 10:40:27.587539 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 104 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140390 v187) v2 ==== 289+0+0 (290492376 0 0) 0x2a51840 con 0x2118420 -13552> 2013-04-03 10:40:27.587614 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140390 v187) v2 from mds.? 10.252.58.31:6800/10908 -13551> 2013-04-03 10:40:27.587626 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140390 v187) v2 -- ?+0 0x20e4340 con 0x2118420 -13550> 2013-04-03 10:40:28.236568 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113553 ==== forward(pg_stats(1 pgs tid 14 v 605) v1) to leader v1 ==== 757+0+0 (40371216 0 0) 0x2a51580 con 0x208a840 -13549> 2013-04-03 10:40:28.236589 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -13548> 2013-04-03 10:40:28.236598 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -13547> 2013-04-03 10:40:28.236648 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 preprocess_query pg_stats(1 pgs tid 14 v 605) v1 from osd.2 10.252.24.30:6801/5656 -13546> 2013-04-03 10:40:28.236672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 prepare_update pg_stats(1 pgs tid 14 v 605) v1 from osd.2 10.252.24.30:6801/5656 -13545> 2013-04-03 10:40:28.236678 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 prepare_pg_stats pg_stats(1 pgs tid 14 v 605) v1 from osd.2 -13544> 2013-04-03 10:40:28.236684 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29053 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -13543> 2013-04-03 10:40:28.286888 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29053 encode_pending v 29054 -13542> 2013-04-03 10:40:28.286927 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427391) queue_proposal bl 654 bytes; ctx = 0x2c3d380 -13541> 2013-04-03 10:40:28.286937 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427326..427391) propose_queued 427392 654 bytes -13540> 2013-04-03 10:40:28.288105 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427391 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787280 -13539> 2013-04-03 10:40:28.315584 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113554 ==== paxos(accept lc 427391 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2126212867 0 0) 0x2787280 con 0x208a840 -13538> 2013-04-03 10:40:28.315882 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427392 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787780 -13537> 2013-04-03 10:40:28.315914 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427392 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2ab4a00 -13536> 2013-04-03 10:40:28.315978 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29053 update_from_paxos applying incremental 29054 -13535> 2013-04-03 10:40:28.315997 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29054 v29054: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2620B/s wr, 0op/s -13534> 2013-04-03 10:40:28.316017 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29054 send_pg_creates to 0 pgs -13533> 2013-04-03 10:40:28.316023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29054 update_logger -13532> 2013-04-03 10:40:28.316033 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29054 create_pending v 29055 -13531> 2013-04-03 10:40:28.316042 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29054 _updated_stats for osd.2 10.252.24.30:6801/5656 -13530> 2013-04-03 10:40:28.316049 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 14) v1 tid 17593) v2 -- ?+0 0x24f8240 con 0x208a840 -13529> 2013-04-03 10:40:28.316059 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29054 check_osd_map already seen 605 >= 605 -13528> 2013-04-03 10:40:28.316062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29054 update_logger -13527> 2013-04-03 10:40:28.316072 7fcdffcdb700 0 log [INF] : pgmap v29054: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2620B/s wr, 0op/s -13526> 2013-04-03 10:40:28.316074 7fcdffcdb700 10 send_log to self -13525> 2013-04-03 10:40:28.316075 7fcdffcdb700 10 log_queue is 2 last_log 120 sent 118 num 2 unsent 2 sending 2 -13524> 2013-04-03 10:40:28.316077 7fcdffcdb700 10 will send 2013-04-03 10:40:22.795431 mon.0 10.252.24.25:6789/0 119 : [INF] pgmap v29053: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 467B/s wr, 0op/s -13523> 2013-04-03 10:40:28.316084 7fcdffcdb700 10 will send 2013-04-03 10:40:28.316073 mon.0 10.252.24.25:6789/0 120 : [INF] pgmap v29054: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2620B/s wr, 0op/s -13522> 2013-04-03 10:40:28.316088 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b76600 -13521> 2013-04-03 10:40:28.316138 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos -13520> 2013-04-03 10:40:28.316157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 -13519> 2013-04-03 10:40:28.316175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13518> 2013-04-03 10:40:28.316197 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76600 con 0x1bac160 -13517> 2013-04-03 10:40:28.316219 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos -13516> 2013-04-03 10:40:28.316233 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 -13515> 2013-04-03 10:40:28.316242 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -13514> 2013-04-03 10:40:28.316256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 preprocess_log log(2 entries) v1 from mon.0 -13513> 2013-04-03 10:40:28.316286 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -13512> 2013-04-03 10:40:28.316298 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 prepare_log log(2 entries) v1 from mon.0 -13511> 2013-04-03 10:40:28.316308 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 logging 2013-04-03 10:40:22.795431 mon.0 10.252.24.25:6789/0 119 : [INF] pgmap v29053: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 467B/s wr, 0op/s -13510> 2013-04-03 10:40:28.316329 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 logging 2013-04-03 10:40:28.316073 mon.0 10.252.24.25:6789/0 120 : [INF] pgmap v29054: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2620B/s wr, 0op/s -13509> 2013-04-03 10:40:28.340623 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113555 ==== paxos(lease_ack lc 427392 fc 427326 pn 0 opn 0) v3 ==== 80+0+0 (2969537434 0 0) 0x2ab4a00 con 0x208a840 -13508> 2013-04-03 10:40:28.340659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos -13507> 2013-04-03 10:40:28.340675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 -13506> 2013-04-03 10:40:28.340690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13505> 2013-04-03 10:40:28.351456 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29054 check_down_pgs -13504> 2013-04-03 10:40:28.351491 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29054 v29054: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2620B/s wr, 0op/s -13503> 2013-04-03 10:40:28.351514 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -13502> 2013-04-03 10:40:28.351536 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -13501> 2013-04-03 10:40:28.351572 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -13500> 2013-04-03 10:40:28.351602 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos -13499> 2013-04-03 10:40:28.351619 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394053 update_from_paxos version 394053 summary v 394053 -13498> 2013-04-03 10:40:28.351628 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394053 log -13497> 2013-04-03 10:40:28.351643 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -13496> 2013-04-03 10:40:28.351658 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -13495> 2013-04-03 10:40:28.485392 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427326..427392) queue_proposal bl 1737 bytes; ctx = 0x2c3d460 -13494> 2013-04-03 10:40:28.541241 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113556 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (424405211 0 0) 0x2995000 con 0x208a840 -13493> 2013-04-03 10:40:28.541256 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -13492> 2013-04-03 10:40:28.541261 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -13491> 2013-04-03 10:40:28.541268 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 -13490> 2013-04-03 10:40:28.541271 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( start_reply flags( reply-to(mon.2 10.252.24.88:6789/0) ) ) v1 -- ?+0 0x2a51580 -13489> 2013-04-03 10:40:28.933715 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113557 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 105 v187) v2) to leader v1 ==== 560+0+0 (246213155 0 0) 0x29952c0 con 0x208a840 -13488> 2013-04-03 10:40:28.933736 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -13487> 2013-04-03 10:40:28.933747 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2995840 from 10.252.24.88:6789/0 -13486> 2013-04-03 10:40:28.933800 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 105 v187) v2 from mds.0 10.252.24.25:6802/3144 -13485> 2013-04-03 10:40:28.933819 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 105 v187) v2 tid 17594) v2 -- ?+0 0x24f8b40 con 0x208a840 -13484> 2013-04-03 10:40:29.315793 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394053 encode_pending v394054 -13483> 2013-04-03 10:40:29.315824 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427326..427392) queue_proposal bl 725 bytes; ctx = 0x2c3d300 -13482> 2013-04-03 10:40:29.315832 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427326..427392) propose_queued 427393 1737 bytes -13481> 2013-04-03 10:40:29.316890 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427392 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787c80 -13480> 2013-04-03 10:40:29.342148 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113558 ==== paxos(accept lc 427392 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (709865664 0 0) 0x2787c80 con 0x208a840 -13479> 2013-04-03 10:40:29.343107 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427393 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4a00 -13478> 2013-04-03 10:40:29.343130 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427393 fc 427326 pn 0 opn 0) v3 -- ?+0 0x2787280 -13477> 2013-04-03 10:40:29.343184 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427393) propose_queued 427394 725 bytes -13476> 2013-04-03 10:40:29.343414 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427393 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4500 -13475> 2013-04-03 10:40:29.376940 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113559 ==== paxos(lease_ack lc 427393 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2834495128 0 0) 0x2ab4500 con 0x208a840 -13474> 2013-04-03 10:40:29.386585 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 104 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21761 v187) v2 ==== 289+0+0 (2217472120 0 0) 0x2ce1dc0 con 0x200d420 -13473> 2013-04-03 10:40:29.386636 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21761 v187) v2 from mds.? 10.252.24.88:6800/17213 -13472> 2013-04-03 10:40:29.386650 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21761 v187) v2 -- ?+0 0x29952c0 con 0x200d420 -13471> 2013-04-03 10:40:29.388625 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113560 ==== paxos(accept lc 427393 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4274455818 0 0) 0x2787280 con 0x208a840 -13470> 2013-04-03 10:40:29.388889 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427394 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2ab4500 -13469> 2013-04-03 10:40:29.388920 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427394 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2787c80 -13468> 2013-04-03 10:40:29.388956 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos -13467> 2013-04-03 10:40:29.388985 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394053 -13466> 2013-04-03 10:40:29.389002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos latest full 393798 -13465> 2013-04-03 10:40:29.389021 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos applying incremental log 394054 2013-04-03 10:40:22.795431 mon.0 10.252.24.25:6789/0 119 : [INF] pgmap v29053: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 467B/s wr, 0op/s -13464> 2013-04-03 10:40:29.389049 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos applying incremental log 394054 2013-04-03 10:40:28.316073 mon.0 10.252.24.25:6789/0 120 : [INF] pgmap v29054: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2620B/s wr, 0op/s -13463> 2013-04-03 10:40:29.389101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 check_subs -13462> 2013-04-03 10:40:29.389135 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 create_pending v 394055 -13461> 2013-04-03 10:40:29.389158 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 _updated_log for mon.0 10.252.24.25:6789/0 -13460> 2013-04-03 10:40:29.389172 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 120) v1 -- ?+0 0x2d98540 con 0x1bac160 -13459> 2013-04-03 10:40:29.389219 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos -13458> 2013-04-03 10:40:29.389235 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394054 -13457> 2013-04-03 10:40:29.389254 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13456> 2013-04-03 10:40:29.389276 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 120) v1 ==== 0+0+0 (0 0 0) 0x2d98540 con 0x1bac160 -13455> 2013-04-03 10:40:29.389285 7fcdffcdb700 10 handle_log_ack log(last 120) v1 -13454> 2013-04-03 10:40:29.389286 7fcdffcdb700 10 logged 2013-04-03 10:40:22.795431 mon.0 10.252.24.25:6789/0 119 : [INF] pgmap v29053: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 467B/s wr, 0op/s -13453> 2013-04-03 10:40:29.389292 7fcdffcdb700 10 logged 2013-04-03 10:40:28.316073 mon.0 10.252.24.25:6789/0 120 : [INF] pgmap v29054: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2620B/s wr, 0op/s -13452> 2013-04-03 10:40:29.401889 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113561 ==== paxos(lease_ack lc 427394 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3470913269 0 0) 0x2787c80 con 0x208a840 -13451> 2013-04-03 10:40:29.401925 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos -13450> 2013-04-03 10:40:29.401941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394054 -13449> 2013-04-03 10:40:29.401958 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13448> 2013-04-03 10:40:30.681202 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2df2c80 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33078/0 -13447> 2013-04-03 10:40:30.681227 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2c80 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33078/0) -13446> 2013-04-03 10:40:30.681244 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2c80 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -13445> 2013-04-03 10:40:30.681451 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2c80 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -13444> 2013-04-03 10:40:31.587657 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 105 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140391 v187) v2 ==== 289+0+0 (842936485 0 0) 0x2a52080 con 0x2118420 -13443> 2013-04-03 10:40:31.587733 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140391 v187) v2 from mds.? 10.252.58.31:6800/10908 -13442> 2013-04-03 10:40:31.587747 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140391 v187) v2 -- ?+0 0x2ce1dc0 con 0x2118420 -13441> 2013-04-03 10:40:32.388998 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427394 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2787a00 -13440> 2013-04-03 10:40:32.389810 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113562 ==== paxos(lease_ack lc 427394 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1841783146 0 0) 0x2787a00 con 0x208a840 -13439> 2013-04-03 10:40:32.389879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos -13438> 2013-04-03 10:40:32.389900 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394054 -13437> 2013-04-03 10:40:32.389919 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13436> 2013-04-03 10:40:32.933955 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113563 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 106 v187) v2) to leader v1 ==== 560+0+0 (2371754562 0 0) 0x2995000 con 0x208a840 -13435> 2013-04-03 10:40:32.933975 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -13434> 2013-04-03 10:40:32.933986 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a84600 from 10.252.24.88:6789/0 -13433> 2013-04-03 10:40:32.934032 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 106 v187) v2 from mds.0 10.252.24.25:6802/3144 -13432> 2013-04-03 10:40:32.934044 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 106 v187) v2 tid 17595) v2 -- ?+0 0x24f8d80 con 0x208a840 -13431> 2013-04-03 10:40:33.237168 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113564 ==== forward(pg_stats(1 pgs tid 15 v 605) v1) to leader v1 ==== 757+0+0 (4227685684 0 0) 0x2a52080 con 0x208a840 -13430> 2013-04-03 10:40:33.237182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -13429> 2013-04-03 10:40:33.237190 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -13428> 2013-04-03 10:40:33.237225 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29054 preprocess_query pg_stats(1 pgs tid 15 v 605) v1 from osd.2 10.252.24.30:6801/5656 -13427> 2013-04-03 10:40:33.237249 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29054 prepare_update pg_stats(1 pgs tid 15 v 605) v1 from osd.2 10.252.24.30:6801/5656 -13426> 2013-04-03 10:40:33.237255 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29054 prepare_pg_stats pg_stats(1 pgs tid 15 v 605) v1 from osd.2 -13425> 2013-04-03 10:40:33.237261 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29054 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -13424> 2013-04-03 10:40:33.287469 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29054 encode_pending v 29055 -13423> 2013-04-03 10:40:33.287508 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427394) queue_proposal bl 654 bytes; ctx = 0x2c3d590 -13422> 2013-04-03 10:40:33.287517 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427394) propose_queued 427395 654 bytes -13421> 2013-04-03 10:40:33.288829 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427394 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787500 -13420> 2013-04-03 10:40:33.314780 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113565 ==== paxos(accept lc 427394 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4109111566 0 0) 0x2787500 con 0x208a840 -13419> 2013-04-03 10:40:33.315129 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427395 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787a00 -13418> 2013-04-03 10:40:33.315162 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427395 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2787c80 -13417> 2013-04-03 10:40:33.315233 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29054 update_from_paxos applying incremental 29055 -13416> 2013-04-03 10:40:33.315253 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 v29055: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1946B/s wr, 0op/s -13415> 2013-04-03 10:40:33.315273 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 send_pg_creates to 0 pgs -13414> 2013-04-03 10:40:33.315277 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 update_logger -13413> 2013-04-03 10:40:33.315287 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 create_pending v 29056 -13412> 2013-04-03 10:40:33.315299 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 _updated_stats for osd.2 10.252.24.30:6801/5656 -13411> 2013-04-03 10:40:33.315305 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 15) v1 tid 17596) v2 -- ?+0 0x24f8b40 con 0x208a840 -13410> 2013-04-03 10:40:33.315315 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 check_osd_map already seen 605 >= 605 -13409> 2013-04-03 10:40:33.315319 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 update_logger -13408> 2013-04-03 10:40:33.315330 7fcdffcdb700 0 log [INF] : pgmap v29055: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1946B/s wr, 0op/s -13407> 2013-04-03 10:40:33.315331 7fcdffcdb700 10 send_log to self -13406> 2013-04-03 10:40:33.315332 7fcdffcdb700 10 log_queue is 1 last_log 121 sent 120 num 1 unsent 1 sending 1 -13405> 2013-04-03 10:40:33.315334 7fcdffcdb700 10 will send 2013-04-03 10:40:33.315331 mon.0 10.252.24.25:6789/0 121 : [INF] pgmap v29055: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1946B/s wr, 0op/s -13404> 2013-04-03 10:40:33.315340 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b76600 -13403> 2013-04-03 10:40:33.315390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos -13402> 2013-04-03 10:40:33.315410 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394054 -13401> 2013-04-03 10:40:33.315429 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13400> 2013-04-03 10:40:33.315449 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76600 con 0x1bac160 -13399> 2013-04-03 10:40:33.315474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos -13398> 2013-04-03 10:40:33.315489 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394054 -13397> 2013-04-03 10:40:33.315498 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13396> 2013-04-03 10:40:33.315510 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 preprocess_log log(1 entries) v1 from mon.0 -13395> 2013-04-03 10:40:33.315534 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13394> 2013-04-03 10:40:33.315547 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 prepare_log log(1 entries) v1 from mon.0 -13393> 2013-04-03 10:40:33.315557 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 logging 2013-04-03 10:40:33.315331 mon.0 10.252.24.25:6789/0 121 : [INF] pgmap v29055: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1946B/s wr, 0op/s -13392> 2013-04-03 10:40:33.339856 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113566 ==== paxos(lease_ack lc 427395 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3381693574 0 0) 0x2787c80 con 0x208a840 -13391> 2013-04-03 10:40:33.339896 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos -13390> 2013-04-03 10:40:33.339913 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394054 -13389> 2013-04-03 10:40:33.339928 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13388> 2013-04-03 10:40:33.351771 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 check_down_pgs -13387> 2013-04-03 10:40:33.351808 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 v29055: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1946B/s wr, 0op/s -13386> 2013-04-03 10:40:33.351832 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -13385> 2013-04-03 10:40:33.351854 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -13384> 2013-04-03 10:40:33.351891 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -13383> 2013-04-03 10:40:33.351922 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos -13382> 2013-04-03 10:40:33.351940 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 update_from_paxos version 394054 summary v 394054 -13381> 2013-04-03 10:40:33.351949 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 log -13380> 2013-04-03 10:40:33.351965 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13379> 2013-04-03 10:40:33.351983 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -13378> 2013-04-03 10:40:33.386688 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 105 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21762 v187) v2 ==== 289+0+0 (3784847871 0 0) 0x2ce2b80 con 0x200d420 -13377> 2013-04-03 10:40:33.386738 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21762 v187) v2 from mds.? 10.252.24.88:6800/17213 -13376> 2013-04-03 10:40:33.386749 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21762 v187) v2 -- ?+0 0x2a52080 con 0x200d420 -13375> 2013-04-03 10:40:34.314989 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394054 encode_pending v394055 -13374> 2013-04-03 10:40:34.315019 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427395) queue_proposal bl 443 bytes; ctx = 0x2c3d490 -13373> 2013-04-03 10:40:34.315027 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427395) propose_queued 427396 443 bytes -13372> 2013-04-03 10:40:34.316080 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427395 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2d26780 -13371> 2013-04-03 10:40:34.338517 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113567 ==== paxos(accept lc 427395 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4284617536 0 0) 0x2d26780 con 0x208a840 -13370> 2013-04-03 10:40:34.339428 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427396 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787c80 -13369> 2013-04-03 10:40:34.339464 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427396 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2787500 -13368> 2013-04-03 10:40:34.339496 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos -13367> 2013-04-03 10:40:34.339522 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos version 394055 summary v 394054 -13366> 2013-04-03 10:40:34.339539 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos latest full 393798 -13365> 2013-04-03 10:40:34.339559 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos applying incremental log 394055 2013-04-03 10:40:33.315331 mon.0 10.252.24.25:6789/0 121 : [INF] pgmap v29055: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1946B/s wr, 0op/s -13364> 2013-04-03 10:40:34.339606 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 check_subs -13363> 2013-04-03 10:40:34.339639 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 create_pending v 394056 -13362> 2013-04-03 10:40:34.339662 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 _updated_log for mon.0 10.252.24.25:6789/0 -13361> 2013-04-03 10:40:34.339675 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 121) v1 -- ?+0 0x2d98380 con 0x1bac160 -13360> 2013-04-03 10:40:34.339728 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos -13359> 2013-04-03 10:40:34.339744 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos version 394055 summary v 394055 -13358> 2013-04-03 10:40:34.339763 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13357> 2013-04-03 10:40:34.339786 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 121) v1 ==== 0+0+0 (0 0 0) 0x2d98380 con 0x1bac160 -13356> 2013-04-03 10:40:34.339794 7fcdffcdb700 10 handle_log_ack log(last 121) v1 -13355> 2013-04-03 10:40:34.339795 7fcdffcdb700 10 logged 2013-04-03 10:40:33.315331 mon.0 10.252.24.25:6789/0 121 : [INF] pgmap v29055: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1946B/s wr, 0op/s -13354> 2013-04-03 10:40:34.351745 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113568 ==== paxos(lease_ack lc 427396 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3884039360 0 0) 0x2787500 con 0x208a840 -13353> 2013-04-03 10:40:34.351781 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos -13352> 2013-04-03 10:40:34.351798 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos version 394055 summary v 394055 -13351> 2013-04-03 10:40:34.351815 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13350> 2013-04-03 10:40:35.587765 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 106 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140392 v187) v2 ==== 289+0+0 (3299770831 0 0) 0x2a51dc0 con 0x2118420 -13349> 2013-04-03 10:40:35.587842 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140392 v187) v2 from mds.? 10.252.58.31:6800/10908 -13348> 2013-04-03 10:40:35.587857 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140392 v187) v2 -- ?+0 0x2ce2b80 con 0x2118420 -13347> 2013-04-03 10:40:36.596141 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113569 ==== forward(pg_stats(1 pgs tid 35 v 605) v1) to leader v1 ==== 753+0+0 (3090370418 0 0) 0x2995580 con 0x208a840 -13346> 2013-04-03 10:40:36.596162 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -13345> 2013-04-03 10:40:36.596173 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -13344> 2013-04-03 10:40:36.596223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 preprocess_query pg_stats(1 pgs tid 35 v 605) v1 from osd.1 10.252.24.25:6805/3572 -13343> 2013-04-03 10:40:36.596248 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 prepare_update pg_stats(1 pgs tid 35 v 605) v1 from osd.1 10.252.24.25:6805/3572 -13342> 2013-04-03 10:40:36.596255 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 prepare_pg_stats pg_stats(1 pgs tid 35 v 605) v1 from osd.1 -13341> 2013-04-03 10:40:36.596261 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -13340> 2013-04-03 10:40:36.646446 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 encode_pending v 29056 -13339> 2013-04-03 10:40:36.646484 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427396) queue_proposal bl 650 bytes; ctx = 0x2c3d430 -13338> 2013-04-03 10:40:36.646493 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427396) propose_queued 427397 650 bytes -13337> 2013-04-03 10:40:36.647461 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427396 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2d26a00 -13336> 2013-04-03 10:40:36.699645 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113570 ==== paxos(accept lc 427396 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1191809995 0 0) 0x2d26a00 con 0x208a840 -13335> 2013-04-03 10:40:36.699972 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427397 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787500 -13334> 2013-04-03 10:40:36.700002 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427397 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2d26780 -13333> 2013-04-03 10:40:36.700067 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29055 update_from_paxos applying incremental 29056 -13332> 2013-04-03 10:40:36.700084 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 v29056: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 979B/s wr, 0op/s -13331> 2013-04-03 10:40:36.700104 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 send_pg_creates to 0 pgs -13330> 2013-04-03 10:40:36.700109 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 update_logger -13329> 2013-04-03 10:40:36.700119 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 create_pending v 29057 -13328> 2013-04-03 10:40:36.700129 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 _updated_stats for osd.1 10.252.24.25:6805/3572 -13327> 2013-04-03 10:40:36.700136 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 35) v1 tid 17597) v2 -- ?+0 0x24f8d80 con 0x208a840 -13326> 2013-04-03 10:40:36.700145 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 check_osd_map already seen 605 >= 605 -13325> 2013-04-03 10:40:36.700149 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 update_logger -13324> 2013-04-03 10:40:36.700166 7fcdffcdb700 0 log [INF] : pgmap v29056: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 979B/s wr, 0op/s -13323> 2013-04-03 10:40:36.700169 7fcdffcdb700 10 send_log to self -13322> 2013-04-03 10:40:36.700170 7fcdffcdb700 10 log_queue is 1 last_log 122 sent 121 num 1 unsent 1 sending 1 -13321> 2013-04-03 10:40:36.700172 7fcdffcdb700 10 will send 2013-04-03 10:40:36.700169 mon.0 10.252.24.25:6789/0 122 : [INF] pgmap v29056: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 979B/s wr, 0op/s -13320> 2013-04-03 10:40:36.700178 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b76600 -13319> 2013-04-03 10:40:36.700228 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos -13318> 2013-04-03 10:40:36.700249 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos version 394055 summary v 394055 -13317> 2013-04-03 10:40:36.700267 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13316> 2013-04-03 10:40:36.700291 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76600 con 0x1bac160 -13315> 2013-04-03 10:40:36.700314 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos -13314> 2013-04-03 10:40:36.700329 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos version 394055 summary v 394055 -13313> 2013-04-03 10:40:36.700339 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13312> 2013-04-03 10:40:36.700351 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 preprocess_log log(1 entries) v1 from mon.0 -13311> 2013-04-03 10:40:36.700375 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13310> 2013-04-03 10:40:36.700387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 prepare_log log(1 entries) v1 from mon.0 -13309> 2013-04-03 10:40:36.700398 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 logging 2013-04-03 10:40:36.700169 mon.0 10.252.24.25:6789/0 122 : [INF] pgmap v29056: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 979B/s wr, 0op/s -13308> 2013-04-03 10:40:36.723595 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113571 ==== paxos(lease_ack lc 427397 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (754334529 0 0) 0x2d26780 con 0x208a840 -13307> 2013-04-03 10:40:36.723638 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos -13306> 2013-04-03 10:40:36.723655 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 update_from_paxos version 394055 summary v 394055 -13305> 2013-04-03 10:40:36.723674 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13304> 2013-04-03 10:40:36.933992 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113572 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 107 v187) v2) to leader v1 ==== 560+0+0 (2980746872 0 0) 0x2996b80 con 0x208a840 -13303> 2013-04-03 10:40:36.934013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -13302> 2013-04-03 10:40:36.934024 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2995b00 from 10.252.24.88:6789/0 -13301> 2013-04-03 10:40:36.934075 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 107 v187) v2 from mds.0 10.252.24.25:6802/3144 -13300> 2013-04-03 10:40:36.934087 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 107 v187) v2 tid 17598) v2 -- ?+0 0x24f8b40 con 0x208a840 -13299> 2013-04-03 10:40:37.386798 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 106 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21763 v187) v2 ==== 289+0+0 (3270938242 0 0) 0x2ce2080 con 0x200d420 -13298> 2013-04-03 10:40:37.386871 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21763 v187) v2 from mds.? 10.252.24.88:6800/17213 -13297> 2013-04-03 10:40:37.386888 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21763 v187) v2 -- ?+0 0x2996b80 con 0x200d420 -13296> 2013-04-03 10:40:37.699829 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394055 encode_pending v394056 -13295> 2013-04-03 10:40:37.699855 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427397) queue_proposal bl 442 bytes; ctx = 0x2c3d440 -13294> 2013-04-03 10:40:37.699865 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427397) propose_queued 427398 442 bytes -13293> 2013-04-03 10:40:37.701560 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427397 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2d26c80 -13292> 2013-04-03 10:40:37.731062 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113573 ==== paxos(accept lc 427397 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4083235468 0 0) 0x2d26c80 con 0x208a840 -13291> 2013-04-03 10:40:37.731397 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427398 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2d26780 -13290> 2013-04-03 10:40:37.731431 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427398 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2d26a00 -13289> 2013-04-03 10:40:37.731468 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos -13288> 2013-04-03 10:40:37.731496 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394055 -13287> 2013-04-03 10:40:37.731511 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos latest full 393798 -13286> 2013-04-03 10:40:37.731530 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos applying incremental log 394056 2013-04-03 10:40:36.700169 mon.0 10.252.24.25:6789/0 122 : [INF] pgmap v29056: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 979B/s wr, 0op/s -13285> 2013-04-03 10:40:37.731576 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 check_subs -13284> 2013-04-03 10:40:37.731608 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 create_pending v 394057 -13283> 2013-04-03 10:40:37.731632 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 _updated_log for mon.0 10.252.24.25:6789/0 -13282> 2013-04-03 10:40:37.731647 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 122) v1 -- ?+0 0x2d981c0 con 0x1bac160 -13281> 2013-04-03 10:40:37.731697 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos -13280> 2013-04-03 10:40:37.731713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 -13279> 2013-04-03 10:40:37.731730 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13278> 2013-04-03 10:40:37.731752 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 122) v1 ==== 0+0+0 (0 0 0) 0x2d981c0 con 0x1bac160 -13277> 2013-04-03 10:40:37.731760 7fcdffcdb700 10 handle_log_ack log(last 122) v1 -13276> 2013-04-03 10:40:37.731761 7fcdffcdb700 10 logged 2013-04-03 10:40:36.700169 mon.0 10.252.24.25:6789/0 122 : [INF] pgmap v29056: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 979B/s wr, 0op/s -13275> 2013-04-03 10:40:37.744296 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113574 ==== paxos(lease_ack lc 427398 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2560948303 0 0) 0x2d26a00 con 0x208a840 -13274> 2013-04-03 10:40:37.744331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos -13273> 2013-04-03 10:40:37.744348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 -13272> 2013-04-03 10:40:37.744364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13271> 2013-04-03 10:40:38.352122 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 check_down_pgs -13270> 2013-04-03 10:40:38.352157 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 v29056: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 979B/s wr, 0op/s -13269> 2013-04-03 10:40:38.352187 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -13268> 2013-04-03 10:40:38.352212 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -13267> 2013-04-03 10:40:38.352249 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -13266> 2013-04-03 10:40:38.352282 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos -13265> 2013-04-03 10:40:38.352301 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 -13264> 2013-04-03 10:40:38.352311 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 log -13263> 2013-04-03 10:40:38.352328 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13262> 2013-04-03 10:40:38.352348 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -13261> 2013-04-03 10:40:39.587909 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 107 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140393 v187) v2 ==== 289+0+0 (3888135858 0 0) 0x2a512c0 con 0x2118420 -13260> 2013-04-03 10:40:39.587987 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140393 v187) v2 from mds.? 10.252.58.31:6800/10908 -13259> 2013-04-03 10:40:39.588002 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140393 v187) v2 -- ?+0 0x2ce2080 con 0x2118420 -13258> 2013-04-03 10:40:40.697187 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2df2a00 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33079/0 -13257> 2013-04-03 10:40:40.697215 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2a00 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33079/0) -13256> 2013-04-03 10:40:40.697231 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2a00 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -13255> 2013-04-03 10:40:40.697433 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2a00 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -13254> 2013-04-03 10:40:40.731458 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427398 fc 427382 pn 0 opn 0) v3 -- ?+0 0x20f4c80 -13253> 2013-04-03 10:40:40.732221 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113575 ==== paxos(lease_ack lc 427398 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2432206401 0 0) 0x20f4c80 con 0x208a840 -13252> 2013-04-03 10:40:40.732296 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos -13251> 2013-04-03 10:40:40.732316 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 -13250> 2013-04-03 10:40:40.732334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13249> 2013-04-03 10:40:40.934110 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113576 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 108 v187) v2) to leader v1 ==== 560+0+0 (583158828 0 0) 0x2995580 con 0x208a840 -13248> 2013-04-03 10:40:40.934134 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -13247> 2013-04-03 10:40:40.934146 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2996600 from 10.252.24.88:6789/0 -13246> 2013-04-03 10:40:40.934199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 108 v187) v2 from mds.0 10.252.24.25:6802/3144 -13245> 2013-04-03 10:40:40.934210 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 108 v187) v2 tid 17599) v2 -- ?+0 0x24f8d80 con 0x208a840 -13244> 2013-04-03 10:40:41.386905 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 107 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21764 v187) v2 ==== 289+0+0 (721400561 0 0) 0x2ce2340 con 0x200d420 -13243> 2013-04-03 10:40:41.386977 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21764 v187) v2 from mds.? 10.252.24.88:6800/17213 -13242> 2013-04-03 10:40:41.386990 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21764 v187) v2 -- ?+0 0x2995580 con 0x200d420 -13241> 2013-04-03 10:40:43.352491 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 v29056: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 979B/s wr, 0op/s -13240> 2013-04-03 10:40:43.352540 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -13239> 2013-04-03 10:40:43.352564 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -13238> 2013-04-03 10:40:43.352610 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -13237> 2013-04-03 10:40:43.352642 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos -13236> 2013-04-03 10:40:43.352662 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 -13235> 2013-04-03 10:40:43.352671 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 log -13234> 2013-04-03 10:40:43.352689 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13233> 2013-04-03 10:40:43.352709 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -13232> 2013-04-03 10:40:43.588009 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 108 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140394 v187) v2 ==== 289+0+0 (2188640053 0 0) 0x2cc5080 con 0x2118420 -13231> 2013-04-03 10:40:43.588060 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140394 v187) v2 from mds.? 10.252.58.31:6800/10908 -13230> 2013-04-03 10:40:43.588073 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140394 v187) v2 -- ?+0 0x2ce2340 con 0x2118420 -13229> 2013-04-03 10:40:43.731521 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427398 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29e0c80 -13228> 2013-04-03 10:40:43.732302 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113577 ==== paxos(lease_ack lc 427398 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2151997480 0 0) 0x29e0c80 con 0x208a840 -13227> 2013-04-03 10:40:43.732349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos -13226> 2013-04-03 10:40:43.732367 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 -13225> 2013-04-03 10:40:43.732384 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13224> 2013-04-03 10:40:44.934248 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113578 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 109 v187) v2) to leader v1 ==== 560+0+0 (3329679710 0 0) 0x2a512c0 con 0x208a840 -13223> 2013-04-03 10:40:44.934270 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -13222> 2013-04-03 10:40:44.934281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29968c0 from 10.252.24.88:6789/0 -13221> 2013-04-03 10:40:44.934334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 109 v187) v2 from mds.0 10.252.24.25:6802/3144 -13220> 2013-04-03 10:40:44.934346 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 109 v187) v2 tid 17600) v2 -- ?+0 0x24f8b40 con 0x208a840 -13219> 2013-04-03 10:40:45.387000 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 108 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21765 v187) v2 ==== 289+0+0 (160501132 0 0) 0x2cc58c0 con 0x200d420 -13218> 2013-04-03 10:40:45.387077 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21765 v187) v2 from mds.? 10.252.24.88:6800/17213 -13217> 2013-04-03 10:40:45.387092 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21765 v187) v2 -- ?+0 0x2a512c0 con 0x200d420 -13216> 2013-04-03 10:40:46.597269 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113579 ==== forward(pg_stats(1 pgs tid 36 v 605) v1) to leader v1 ==== 753+0+0 (1557649939 0 0) 0x2cc5080 con 0x208a840 -13215> 2013-04-03 10:40:46.597288 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -13214> 2013-04-03 10:40:46.597299 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -13213> 2013-04-03 10:40:46.597349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 preprocess_query pg_stats(1 pgs tid 36 v 605) v1 from osd.1 10.252.24.25:6805/3572 -13212> 2013-04-03 10:40:46.597374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 prepare_update pg_stats(1 pgs tid 36 v 605) v1 from osd.1 10.252.24.25:6805/3572 -13211> 2013-04-03 10:40:46.597381 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 prepare_pg_stats pg_stats(1 pgs tid 36 v 605) v1 from osd.1 -13210> 2013-04-03 10:40:46.597387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -13209> 2013-04-03 10:40:46.647585 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 encode_pending v 29057 -13208> 2013-04-03 10:40:46.647621 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427398) queue_proposal bl 650 bytes; ctx = 0x2c3d570 -13207> 2013-04-03 10:40:46.647629 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427398) propose_queued 427399 650 bytes -13206> 2013-04-03 10:40:46.652103 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427398 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0a00 -13205> 2013-04-03 10:40:46.721189 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113580 ==== paxos(accept lc 427398 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3279303199 0 0) 0x29e0a00 con 0x208a840 -13204> 2013-04-03 10:40:46.721561 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427399 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0c80 -13203> 2013-04-03 10:40:46.721588 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427399 fc 427382 pn 0 opn 0) v3 -- ?+0 0x20f4c80 -13202> 2013-04-03 10:40:46.721659 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29056 update_from_paxos applying incremental 29057 -13201> 2013-04-03 10:40:46.721678 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 v29057: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 613B/s wr, 0op/s -13200> 2013-04-03 10:40:46.721698 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 send_pg_creates to 0 pgs -13199> 2013-04-03 10:40:46.721702 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 update_logger -13198> 2013-04-03 10:40:46.721713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 create_pending v 29058 -13197> 2013-04-03 10:40:46.721722 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 _updated_stats for osd.1 10.252.24.25:6805/3572 -13196> 2013-04-03 10:40:46.721730 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 36) v1 tid 17601) v2 -- ?+0 0x24f8d80 con 0x208a840 -13195> 2013-04-03 10:40:46.721740 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 check_osd_map already seen 605 >= 605 -13194> 2013-04-03 10:40:46.721744 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 update_logger -13193> 2013-04-03 10:40:46.721755 7fcdffcdb700 0 log [INF] : pgmap v29057: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 613B/s wr, 0op/s -13192> 2013-04-03 10:40:46.721756 7fcdffcdb700 10 send_log to self -13191> 2013-04-03 10:40:46.721757 7fcdffcdb700 10 log_queue is 1 last_log 123 sent 122 num 1 unsent 1 sending 1 -13190> 2013-04-03 10:40:46.721759 7fcdffcdb700 10 will send 2013-04-03 10:40:46.721756 mon.0 10.252.24.25:6789/0 123 : [INF] pgmap v29057: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 613B/s wr, 0op/s -13189> 2013-04-03 10:40:46.721765 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b76600 -13188> 2013-04-03 10:40:46.721823 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos -13187> 2013-04-03 10:40:46.721844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 -13186> 2013-04-03 10:40:46.721863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13185> 2013-04-03 10:40:46.721885 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76600 con 0x1bac160 -13184> 2013-04-03 10:40:46.721907 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos -13183> 2013-04-03 10:40:46.721923 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 -13182> 2013-04-03 10:40:46.721932 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13181> 2013-04-03 10:40:46.721944 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 preprocess_log log(1 entries) v1 from mon.0 -13180> 2013-04-03 10:40:46.721968 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13179> 2013-04-03 10:40:46.721980 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 prepare_log log(1 entries) v1 from mon.0 -13178> 2013-04-03 10:40:46.721991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 logging 2013-04-03 10:40:46.721756 mon.0 10.252.24.25:6789/0 123 : [INF] pgmap v29057: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 613B/s wr, 0op/s -13177> 2013-04-03 10:40:46.747268 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113581 ==== paxos(lease_ack lc 427399 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (228661502 0 0) 0x20f4c80 con 0x208a840 -13176> 2013-04-03 10:40:46.747310 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos -13175> 2013-04-03 10:40:46.747327 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 update_from_paxos version 394056 summary v 394056 -13174> 2013-04-03 10:40:46.747342 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13173> 2013-04-03 10:40:46.762784 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113582 ==== forward(pg_stats(1 pgs tid 30450 v 605) v1) to leader v1 ==== 757+0+0 (248958844 0 0) 0x2996080 con 0x208a840 -13172> 2013-04-03 10:40:46.762797 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -13171> 2013-04-03 10:40:46.762805 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -13170> 2013-04-03 10:40:46.762834 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 preprocess_query pg_stats(1 pgs tid 30450 v 605) v1 from osd.3 10.252.24.88:6801/17538 -13169> 2013-04-03 10:40:46.762855 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 prepare_update pg_stats(1 pgs tid 30450 v 605) v1 from osd.3 10.252.24.88:6801/17538 -13168> 2013-04-03 10:40:46.762861 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 prepare_pg_stats pg_stats(1 pgs tid 30450 v 605) v1 from osd.3 -13167> 2013-04-03 10:40:46.762867 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -13166> 2013-04-03 10:40:47.588133 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 109 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140395 v187) v2 ==== 289+0+0 (2703048776 0 0) 0x2cc4580 con 0x2118420 -13165> 2013-04-03 10:40:47.588209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140395 v187) v2 from mds.? 10.252.58.31:6800/10908 -13164> 2013-04-03 10:40:47.588223 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140395 v187) v2 -- ?+0 0x2996080 con 0x2118420 -13163> 2013-04-03 10:40:47.721381 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394056 encode_pending v394057 -13162> 2013-04-03 10:40:47.721410 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427399) queue_proposal bl 442 bytes; ctx = 0x2c3d450 -13161> 2013-04-03 10:40:47.721417 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427399) propose_queued 427400 442 bytes -13160> 2013-04-03 10:40:47.722716 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427399 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4280 -13159> 2013-04-03 10:40:47.722810 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 encode_pending v 29058 -13158> 2013-04-03 10:40:47.722843 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427382..427399) queue_proposal bl 654 bytes; ctx = 0x2c3d410 -13157> 2013-04-03 10:40:47.722850 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427382..427399) propose_new_value not active; proposal queued -13156> 2013-04-03 10:40:47.755630 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113583 ==== paxos(accept lc 427399 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2528509045 0 0) 0x29b4280 con 0x208a840 -13155> 2013-04-03 10:40:47.760684 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427400 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4c80 -13154> 2013-04-03 10:40:47.760709 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427400 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29e0a00 -13153> 2013-04-03 10:40:47.760748 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos -13152> 2013-04-03 10:40:47.760775 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos version 394057 summary v 394056 -13151> 2013-04-03 10:40:47.760794 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos latest full 393798 -13150> 2013-04-03 10:40:47.760813 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos applying incremental log 394057 2013-04-03 10:40:46.721756 mon.0 10.252.24.25:6789/0 123 : [INF] pgmap v29057: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 613B/s wr, 0op/s -13149> 2013-04-03 10:40:47.760859 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 check_subs -13148> 2013-04-03 10:40:47.760891 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 create_pending v 394058 -13147> 2013-04-03 10:40:47.760914 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 _updated_log for mon.0 10.252.24.25:6789/0 -13146> 2013-04-03 10:40:47.760929 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 123) v1 -- ?+0 0x2d98e00 con 0x1bac160 -13145> 2013-04-03 10:40:47.760954 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427400) propose_queued 427401 654 bytes -13144> 2013-04-03 10:40:47.761168 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427400 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2d26a00 -13143> 2013-04-03 10:40:47.761201 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 123) v1 ==== 0+0+0 (0 0 0) 0x2d98e00 con 0x1bac160 -13142> 2013-04-03 10:40:47.761218 7fcdffcdb700 10 handle_log_ack log(last 123) v1 -13141> 2013-04-03 10:40:47.761221 7fcdffcdb700 10 logged 2013-04-03 10:40:46.721756 mon.0 10.252.24.25:6789/0 123 : [INF] pgmap v29057: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 613B/s wr, 0op/s -13140> 2013-04-03 10:40:47.781787 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113584 ==== paxos(lease_ack lc 427400 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1320350591 0 0) 0x2d26a00 con 0x208a840 -13139> 2013-04-03 10:40:47.794029 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113585 ==== paxos(accept lc 427400 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (283743941 0 0) 0x29e0a00 con 0x208a840 -13138> 2013-04-03 10:40:47.794314 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427401 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2d26a00 -13137> 2013-04-03 10:40:47.794345 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427401 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29b4280 -13136> 2013-04-03 10:40:47.794416 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29057 update_from_paxos applying incremental 29058 -13135> 2013-04-03 10:40:47.794436 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 v29058: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 462B/s wr, 0op/s -13134> 2013-04-03 10:40:47.794456 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 send_pg_creates to 0 pgs -13133> 2013-04-03 10:40:47.794459 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 update_logger -13132> 2013-04-03 10:40:47.794474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 create_pending v 29059 -13131> 2013-04-03 10:40:47.794486 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 _updated_stats for osd.3 10.252.24.88:6801/17538 -13130> 2013-04-03 10:40:47.794493 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30450) v1 tid 17602) v2 -- ?+0 0x24f8b40 con 0x208a840 -13129> 2013-04-03 10:40:47.794503 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 check_osd_map already seen 605 >= 605 -13128> 2013-04-03 10:40:47.794506 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 update_logger -13127> 2013-04-03 10:40:47.794516 7fcdffcdb700 0 log [INF] : pgmap v29058: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 462B/s wr, 0op/s -13126> 2013-04-03 10:40:47.794518 7fcdffcdb700 10 send_log to self -13125> 2013-04-03 10:40:47.794519 7fcdffcdb700 10 log_queue is 1 last_log 124 sent 123 num 1 unsent 1 sending 1 -13124> 2013-04-03 10:40:47.794520 7fcdffcdb700 10 will send 2013-04-03 10:40:47.794517 mon.0 10.252.24.25:6789/0 124 : [INF] pgmap v29058: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 462B/s wr, 0op/s -13123> 2013-04-03 10:40:47.794527 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75400 -13122> 2013-04-03 10:40:47.794572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos -13121> 2013-04-03 10:40:47.794590 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos version 394057 summary v 394057 -13120> 2013-04-03 10:40:47.794608 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13119> 2013-04-03 10:40:47.794633 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75400 con 0x1bac160 -13118> 2013-04-03 10:40:47.794655 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos -13117> 2013-04-03 10:40:47.794671 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos version 394057 summary v 394057 -13116> 2013-04-03 10:40:47.794680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13115> 2013-04-03 10:40:47.794692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 preprocess_log log(1 entries) v1 from mon.0 -13114> 2013-04-03 10:40:47.794711 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13113> 2013-04-03 10:40:47.794728 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 prepare_log log(1 entries) v1 from mon.0 -13112> 2013-04-03 10:40:47.794738 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 logging 2013-04-03 10:40:47.794517 mon.0 10.252.24.25:6789/0 124 : [INF] pgmap v29058: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 462B/s wr, 0op/s -13111> 2013-04-03 10:40:47.854619 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113586 ==== paxos(lease_ack lc 427401 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3032840918 0 0) 0x29b4280 con 0x208a840 -13110> 2013-04-03 10:40:47.854660 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos -13109> 2013-04-03 10:40:47.854677 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos version 394057 summary v 394057 -13108> 2013-04-03 10:40:47.854692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13107> 2013-04-03 10:40:48.352852 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 check_down_pgs -13106> 2013-04-03 10:40:48.352889 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 v29058: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 462B/s wr, 0op/s -13105> 2013-04-03 10:40:48.352922 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -13104> 2013-04-03 10:40:48.352947 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -13103> 2013-04-03 10:40:48.352984 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -13102> 2013-04-03 10:40:48.353016 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos -13101> 2013-04-03 10:40:48.353036 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 update_from_paxos version 394057 summary v 394057 -13100> 2013-04-03 10:40:48.353046 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 log -13099> 2013-04-03 10:40:48.353063 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13098> 2013-04-03 10:40:48.353083 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -13097> 2013-04-03 10:40:48.794211 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394057 encode_pending v394058 -13096> 2013-04-03 10:40:48.794242 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427401) queue_proposal bl 442 bytes; ctx = 0x2c3d580 -13095> 2013-04-03 10:40:48.794251 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427401) propose_queued 427402 442 bytes -13094> 2013-04-03 10:40:48.795462 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427401 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4000 -13093> 2013-04-03 10:40:48.821307 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113587 ==== paxos(accept lc 427401 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1427631610 0 0) 0x29b4000 con 0x208a840 -13092> 2013-04-03 10:40:48.821639 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427402 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4280 -13091> 2013-04-03 10:40:48.821661 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427402 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29e0a00 -13090> 2013-04-03 10:40:48.821698 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos -13089> 2013-04-03 10:40:48.821726 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394057 -13088> 2013-04-03 10:40:48.821742 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos latest full 393798 -13087> 2013-04-03 10:40:48.821762 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos applying incremental log 394058 2013-04-03 10:40:47.794517 mon.0 10.252.24.25:6789/0 124 : [INF] pgmap v29058: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 462B/s wr, 0op/s -13086> 2013-04-03 10:40:48.821807 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 check_subs -13085> 2013-04-03 10:40:48.821847 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 create_pending v 394059 -13084> 2013-04-03 10:40:48.821871 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 _updated_log for mon.0 10.252.24.25:6789/0 -13083> 2013-04-03 10:40:48.821884 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 124) v1 -- ?+0 0x2d98e00 con 0x1bac160 -13082> 2013-04-03 10:40:48.821931 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos -13081> 2013-04-03 10:40:48.821947 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394058 -13080> 2013-04-03 10:40:48.821968 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13079> 2013-04-03 10:40:48.821990 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 124) v1 ==== 0+0+0 (0 0 0) 0x2d98e00 con 0x1bac160 -13078> 2013-04-03 10:40:48.821998 7fcdffcdb700 10 handle_log_ack log(last 124) v1 -13077> 2013-04-03 10:40:48.822000 7fcdffcdb700 10 logged 2013-04-03 10:40:47.794517 mon.0 10.252.24.25:6789/0 124 : [INF] pgmap v29058: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 462B/s wr, 0op/s -13076> 2013-04-03 10:40:48.834980 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113588 ==== paxos(lease_ack lc 427402 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (935244079 0 0) 0x29e0a00 con 0x208a840 -13075> 2013-04-03 10:40:48.835018 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos -13074> 2013-04-03 10:40:48.835035 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394058 -13073> 2013-04-03 10:40:48.835052 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13072> 2013-04-03 10:40:48.934340 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113589 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 110 v187) v2) to leader v1 ==== 560+0+0 (3719070555 0 0) 0x2a832c0 con 0x208a840 -13071> 2013-04-03 10:40:48.934355 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -13070> 2013-04-03 10:40:48.934364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83000 from 10.252.24.88:6789/0 -13069> 2013-04-03 10:40:48.934397 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 110 v187) v2 from mds.0 10.252.24.25:6802/3144 -13068> 2013-04-03 10:40:48.934408 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 110 v187) v2 tid 17603) v2 -- ?+0 0x24f8d80 con 0x208a840 -13067> 2013-04-03 10:40:49.387116 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 109 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21766 v187) v2 ==== 289+0+0 (1814219787 0 0) 0x2cc5340 con 0x200d420 -13066> 2013-04-03 10:40:49.387193 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21766 v187) v2 from mds.? 10.252.24.88:6800/17213 -13065> 2013-04-03 10:40:49.387211 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21766 v187) v2 -- ?+0 0x2a832c0 con 0x200d420 -13064> 2013-04-03 10:40:50.713190 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2df2780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33080/0 -13063> 2013-04-03 10:40:50.713219 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33080/0) -13062> 2013-04-03 10:40:50.713233 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2780 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -13061> 2013-04-03 10:40:50.713438 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2780 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -13060> 2013-04-03 10:40:51.588270 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 110 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140396 v187) v2 ==== 289+0+0 (1226079291 0 0) 0x2cc5600 con 0x2118420 -13059> 2013-04-03 10:40:51.588343 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140396 v187) v2 from mds.? 10.252.58.31:6800/10908 -13058> 2013-04-03 10:40:51.588357 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140396 v187) v2 -- ?+0 0x2cc5340 con 0x2118420 -13057> 2013-04-03 10:40:51.821731 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427402 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29fd500 -13056> 2013-04-03 10:40:51.822496 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113590 ==== paxos(lease_ack lc 427402 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1933757478 0 0) 0x29fd500 con 0x208a840 -13055> 2013-04-03 10:40:51.822542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos -13054> 2013-04-03 10:40:51.822563 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394058 -13053> 2013-04-03 10:40:51.822580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13052> 2013-04-03 10:40:52.934527 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113591 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 111 v187) v2) to leader v1 ==== 560+0+0 (2037423365 0 0) 0x2cc4580 con 0x208a840 -13051> 2013-04-03 10:40:52.934547 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -13050> 2013-04-03 10:40:52.934557 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83dc0 from 10.252.24.88:6789/0 -13049> 2013-04-03 10:40:52.934611 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 111 v187) v2 from mds.0 10.252.24.25:6802/3144 -13048> 2013-04-03 10:40:52.934623 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 111 v187) v2 tid 17604) v2 -- ?+0 0x24f8b40 con 0x208a840 -13047> 2013-04-03 10:40:53.239723 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113592 ==== forward(pg_stats(1 pgs tid 16 v 605) v1) to leader v1 ==== 757+0+0 (244651286 0 0) 0x2cc5600 con 0x208a840 -13046> 2013-04-03 10:40:53.239750 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -13045> 2013-04-03 10:40:53.239761 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -13044> 2013-04-03 10:40:53.239810 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 preprocess_query pg_stats(1 pgs tid 16 v 605) v1 from osd.2 10.252.24.30:6801/5656 -13043> 2013-04-03 10:40:53.239835 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 prepare_update pg_stats(1 pgs tid 16 v 605) v1 from osd.2 10.252.24.30:6801/5656 -13042> 2013-04-03 10:40:53.239841 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 prepare_pg_stats pg_stats(1 pgs tid 16 v 605) v1 from osd.2 -13041> 2013-04-03 10:40:53.239851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -13040> 2013-04-03 10:40:53.290070 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 encode_pending v 29059 -13039> 2013-04-03 10:40:53.290460 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427402) queue_proposal bl 654 bytes; ctx = 0x2c3d5a0 -13038> 2013-04-03 10:40:53.290472 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427402) propose_queued 427403 654 bytes -13037> 2013-04-03 10:40:53.291713 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427402 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd000 -13036> 2013-04-03 10:40:53.317215 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113593 ==== paxos(accept lc 427402 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2201462862 0 0) 0x29fd000 con 0x208a840 -13035> 2013-04-03 10:40:53.335999 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427403 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd500 -13034> 2013-04-03 10:40:53.336028 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427403 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29e0a00 -13033> 2013-04-03 10:40:53.336107 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29058 update_from_paxos applying incremental 29059 -13032> 2013-04-03 10:40:53.336128 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 v29059: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3237B/s wr, 0op/s -13031> 2013-04-03 10:40:53.336149 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 send_pg_creates to 0 pgs -13030> 2013-04-03 10:40:53.336154 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 update_logger -13029> 2013-04-03 10:40:53.336164 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 create_pending v 29060 -13028> 2013-04-03 10:40:53.336174 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 _updated_stats for osd.2 10.252.24.30:6801/5656 -13027> 2013-04-03 10:40:53.336181 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 16) v1 tid 17605) v2 -- ?+0 0x24f8d80 con 0x208a840 -13026> 2013-04-03 10:40:53.336192 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 check_osd_map already seen 605 >= 605 -13025> 2013-04-03 10:40:53.336194 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 update_logger -13024> 2013-04-03 10:40:53.336205 7fcdffcdb700 0 log [INF] : pgmap v29059: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3237B/s wr, 0op/s -13023> 2013-04-03 10:40:53.336206 7fcdffcdb700 10 send_log to self -13022> 2013-04-03 10:40:53.336207 7fcdffcdb700 10 log_queue is 1 last_log 125 sent 124 num 1 unsent 1 sending 1 -13021> 2013-04-03 10:40:53.336209 7fcdffcdb700 10 will send 2013-04-03 10:40:53.336206 mon.0 10.252.24.25:6789/0 125 : [INF] pgmap v29059: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3237B/s wr, 0op/s -13020> 2013-04-03 10:40:53.336215 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75400 -13019> 2013-04-03 10:40:53.336265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos -13018> 2013-04-03 10:40:53.336288 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394058 -13017> 2013-04-03 10:40:53.336308 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13016> 2013-04-03 10:40:53.336330 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75400 con 0x1bac160 -13015> 2013-04-03 10:40:53.336352 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos -13014> 2013-04-03 10:40:53.336367 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394058 -13013> 2013-04-03 10:40:53.336376 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13012> 2013-04-03 10:40:53.336389 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 preprocess_log log(1 entries) v1 from mon.0 -13011> 2013-04-03 10:40:53.336413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -13010> 2013-04-03 10:40:53.336426 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 prepare_log log(1 entries) v1 from mon.0 -13009> 2013-04-03 10:40:53.336436 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 logging 2013-04-03 10:40:53.336206 mon.0 10.252.24.25:6789/0 125 : [INF] pgmap v29059: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3237B/s wr, 0op/s -13008> 2013-04-03 10:40:53.346213 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113594 ==== paxos(lease_ack lc 427403 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1512180358 0 0) 0x29e0a00 con 0x208a840 -13007> 2013-04-03 10:40:53.346254 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos -13006> 2013-04-03 10:40:53.346271 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394058 -13005> 2013-04-03 10:40:53.346286 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -13004> 2013-04-03 10:40:53.353182 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 check_down_pgs -13003> 2013-04-03 10:40:53.353211 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 v29059: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3237B/s wr, 0op/s -13002> 2013-04-03 10:40:53.353242 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -13001> 2013-04-03 10:40:53.353264 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -13000> 2013-04-03 10:40:53.353300 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -12999> 2013-04-03 10:40:53.353339 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos -12998> 2013-04-03 10:40:53.353357 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 update_from_paxos version 394058 summary v 394058 -12997> 2013-04-03 10:40:53.353366 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 log -12996> 2013-04-03 10:40:53.353383 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12995> 2013-04-03 10:40:53.353399 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -12994> 2013-04-03 10:40:53.387181 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 110 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21767 v187) v2 ==== 289+0+0 (1330410358 0 0) 0x2cc4dc0 con 0x200d420 -12993> 2013-04-03 10:40:53.387229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21767 v187) v2 from mds.? 10.252.24.88:6800/17213 -12992> 2013-04-03 10:40:53.387241 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21767 v187) v2 -- ?+0 0x2cc5600 con 0x200d420 -12991> 2013-04-03 10:40:54.317431 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394058 encode_pending v394059 -12990> 2013-04-03 10:40:54.317461 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427403) queue_proposal bl 443 bytes; ctx = 0x2c3d420 -12989> 2013-04-03 10:40:54.317469 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427403) propose_queued 427404 443 bytes -12988> 2013-04-03 10:40:54.318583 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427403 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd780 -12987> 2013-04-03 10:40:54.346316 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113595 ==== paxos(accept lc 427403 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (431641192 0 0) 0x29fd780 con 0x208a840 -12986> 2013-04-03 10:40:54.346601 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427404 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29e0a00 -12985> 2013-04-03 10:40:54.346636 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427404 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29fd000 -12984> 2013-04-03 10:40:54.346677 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos -12983> 2013-04-03 10:40:54.346705 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos version 394059 summary v 394058 -12982> 2013-04-03 10:40:54.346719 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos latest full 393798 -12981> 2013-04-03 10:40:54.346738 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos applying incremental log 394059 2013-04-03 10:40:53.336206 mon.0 10.252.24.25:6789/0 125 : [INF] pgmap v29059: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3237B/s wr, 0op/s -12980> 2013-04-03 10:40:54.346785 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 check_subs -12979> 2013-04-03 10:40:54.346817 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 create_pending v 394060 -12978> 2013-04-03 10:40:54.346841 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 _updated_log for mon.0 10.252.24.25:6789/0 -12977> 2013-04-03 10:40:54.346854 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 125) v1 -- ?+0 0x28a5a80 con 0x1bac160 -12976> 2013-04-03 10:40:54.346906 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos -12975> 2013-04-03 10:40:54.346923 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos version 394059 summary v 394059 -12974> 2013-04-03 10:40:54.346941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12973> 2013-04-03 10:40:54.346964 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 125) v1 ==== 0+0+0 (0 0 0) 0x28a5a80 con 0x1bac160 -12972> 2013-04-03 10:40:54.346973 7fcdffcdb700 10 handle_log_ack log(last 125) v1 -12971> 2013-04-03 10:40:54.346974 7fcdffcdb700 10 logged 2013-04-03 10:40:53.336206 mon.0 10.252.24.25:6789/0 125 : [INF] pgmap v29059: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3237B/s wr, 0op/s -12970> 2013-04-03 10:40:54.358682 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113596 ==== paxos(lease_ack lc 427404 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3938407132 0 0) 0x29fd000 con 0x208a840 -12969> 2013-04-03 10:40:54.358717 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos -12968> 2013-04-03 10:40:54.358734 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos version 394059 summary v 394059 -12967> 2013-04-03 10:40:54.358751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12966> 2013-04-03 10:40:55.588394 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 111 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140397 v187) v2 ==== 289+0+0 (1786430278 0 0) 0x2cc5b80 con 0x2118420 -12965> 2013-04-03 10:40:55.588472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140397 v187) v2 from mds.? 10.252.58.31:6800/10908 -12964> 2013-04-03 10:40:55.588487 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140397 v187) v2 -- ?+0 0x2cc4dc0 con 0x2118420 -12963> 2013-04-03 10:40:56.934654 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113597 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 112 v187) v2) to leader v1 ==== 560+0+0 (1568170169 0 0) 0x2a84b80 con 0x208a840 -12962> 2013-04-03 10:40:56.934676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -12961> 2013-04-03 10:40:56.934687 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83580 from 10.252.24.88:6789/0 -12960> 2013-04-03 10:40:56.934741 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 112 v187) v2 from mds.0 10.252.24.25:6802/3144 -12959> 2013-04-03 10:40:56.934753 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 112 v187) v2 tid 17606) v2 -- ?+0 0x24f8b40 con 0x208a840 -12958> 2013-04-03 10:40:57.346711 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427404 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29fda00 -12957> 2013-04-03 10:40:57.347643 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113598 ==== paxos(lease_ack lc 427404 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2874163644 0 0) 0x29fda00 con 0x208a840 -12956> 2013-04-03 10:40:57.347717 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos -12955> 2013-04-03 10:40:57.347738 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos version 394059 summary v 394059 -12954> 2013-04-03 10:40:57.347756 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12953> 2013-04-03 10:40:57.387285 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 111 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21768 v187) v2 ==== 289+0+0 (3118252572 0 0) 0x2cc42c0 con 0x200d420 -12952> 2013-04-03 10:40:57.387330 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21768 v187) v2 from mds.? 10.252.24.88:6800/17213 -12951> 2013-04-03 10:40:57.387342 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21768 v187) v2 -- ?+0 0x2a84b80 con 0x200d420 -12950> 2013-04-03 10:40:58.353542 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29059 v29059: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 3237B/s wr, 0op/s -12949> 2013-04-03 10:40:58.353590 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -12948> 2013-04-03 10:40:58.353614 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -12947> 2013-04-03 10:40:58.353660 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -12946> 2013-04-03 10:40:58.353691 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos -12945> 2013-04-03 10:40:58.353996 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 update_from_paxos version 394059 summary v 394059 -12944> 2013-04-03 10:40:58.354012 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394059 log -12943> 2013-04-03 10:40:58.354030 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12942> 2013-04-03 10:40:58.354050 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -12941> 2013-04-03 10:40:58.541332 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) -12940> 2013-04-03 10:40:58.541361 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader -12939> 2013-04-03 10:40:59.588504 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 112 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140398 v187) v2 ==== 289+0+0 (264831681 0 0) 0x2cc4000 con 0x2118420 -12938> 2013-04-03 10:40:59.588575 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140398 v187) v2 from mds.? 10.252.58.31:6800/10908 -12937> 2013-04-03 10:40:59.588588 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140398 v187) v2 -- ?+0 0x2cc42c0 con 0x2118420 -12936> 2013-04-03 10:41:00.346785 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427404 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2c6ea00 -12935> 2013-04-03 10:41:00.347603 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113599 ==== paxos(lease_ack lc 427404 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (38101092 0 0) 0x2c6ea00 con 0x208a840 -12934> 2013-04-03 10:41:00.347681 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos -12933> 2013-04-03 10:41:00.347701 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos version 394059 summary v 394059 -12932> 2013-04-03 10:41:00.347718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12931> 2013-04-03 10:41:00.934697 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113600 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 113 v187) v2) to leader v1 ==== 560+0+0 (1412546234 0 0) 0x2cc5b80 con 0x208a840 -12930> 2013-04-03 10:41:00.934713 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -12929> 2013-04-03 10:41:00.934721 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83b00 from 10.252.24.88:6789/0 -12928> 2013-04-03 10:41:00.934756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 113 v187) v2 from mds.0 10.252.24.25:6802/3144 -12927> 2013-04-03 10:41:00.934767 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 113 v187) v2 tid 17607) v2 -- ?+0 0x24f8d80 con 0x208a840 -12926> 2013-04-03 10:41:01.387438 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 112 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21769 v187) v2 ==== 289+0+0 (2595388769 0 0) 0x2cc4b00 con 0x200d420 -12925> 2013-04-03 10:41:01.387515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21769 v187) v2 from mds.? 10.252.24.88:6800/17213 -12924> 2013-04-03 10:41:01.387528 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21769 v187) v2 -- ?+0 0x2cc5b80 con 0x200d420 -12923> 2013-04-03 10:41:01.598972 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113601 ==== forward(pg_stats(1 pgs tid 37 v 605) v1) to leader v1 ==== 753+0+0 (988322451 0 0) 0x2cc4000 con 0x208a840 -12922> 2013-04-03 10:41:01.598985 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -12921> 2013-04-03 10:41:01.598994 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -12920> 2013-04-03 10:41:01.599027 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29059 preprocess_query pg_stats(1 pgs tid 37 v 605) v1 from osd.1 10.252.24.25:6805/3572 -12919> 2013-04-03 10:41:01.599051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29059 prepare_update pg_stats(1 pgs tid 37 v 605) v1 from osd.1 10.252.24.25:6805/3572 -12918> 2013-04-03 10:41:01.599057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29059 prepare_pg_stats pg_stats(1 pgs tid 37 v 605) v1 from osd.1 -12917> 2013-04-03 10:41:01.599062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29059 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -12916> 2013-04-03 10:41:01.649248 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29059 encode_pending v 29060 -12915> 2013-04-03 10:41:01.649284 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427404) queue_proposal bl 650 bytes; ctx = 0x2c3d540 -12914> 2013-04-03 10:41:01.649295 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427404) propose_queued 427405 650 bytes -12913> 2013-04-03 10:41:01.657775 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427404 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6e000 -12912> 2013-04-03 10:41:01.816826 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113602 ==== paxos(accept lc 427404 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1732336231 0 0) 0x2c6e000 con 0x208a840 -12911> 2013-04-03 10:41:01.817148 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427405 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6ea00 -12910> 2013-04-03 10:41:01.817178 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427405 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29fda00 -12909> 2013-04-03 10:41:01.817252 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29059 update_from_paxos applying incremental 29060 -12908> 2013-04-03 10:41:01.817268 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 v29060: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1764B/s wr, 0op/s -12907> 2013-04-03 10:41:01.817287 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 send_pg_creates to 0 pgs -12906> 2013-04-03 10:41:01.817290 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 update_logger -12905> 2013-04-03 10:41:01.817299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 create_pending v 29061 -12904> 2013-04-03 10:41:01.817308 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29060 _updated_stats for osd.1 10.252.24.25:6805/3572 -12903> 2013-04-03 10:41:01.817315 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 37) v1 tid 17608) v2 -- ?+0 0x24f8b40 con 0x208a840 -12902> 2013-04-03 10:41:01.817324 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 check_osd_map already seen 605 >= 605 -12901> 2013-04-03 10:41:01.817328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 update_logger -12900> 2013-04-03 10:41:01.817338 7fcdffcdb700 0 log [INF] : pgmap v29060: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1764B/s wr, 0op/s -12899> 2013-04-03 10:41:01.817339 7fcdffcdb700 10 send_log to self -12898> 2013-04-03 10:41:01.817340 7fcdffcdb700 10 log_queue is 1 last_log 126 sent 125 num 1 unsent 1 sending 1 -12897> 2013-04-03 10:41:01.817342 7fcdffcdb700 10 will send 2013-04-03 10:41:01.817339 mon.0 10.252.24.25:6789/0 126 : [INF] pgmap v29060: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1764B/s wr, 0op/s -12896> 2013-04-03 10:41:01.817348 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75400 -12895> 2013-04-03 10:41:01.817396 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos -12894> 2013-04-03 10:41:01.817416 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos version 394059 summary v 394059 -12893> 2013-04-03 10:41:01.817434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12892> 2013-04-03 10:41:01.817457 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75400 con 0x1bac160 -12891> 2013-04-03 10:41:01.817479 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos -12890> 2013-04-03 10:41:01.817494 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos version 394059 summary v 394059 -12889> 2013-04-03 10:41:01.817503 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12888> 2013-04-03 10:41:01.817514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 preprocess_log log(1 entries) v1 from mon.0 -12887> 2013-04-03 10:41:01.817538 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12886> 2013-04-03 10:41:01.817551 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 prepare_log log(1 entries) v1 from mon.0 -12885> 2013-04-03 10:41:01.817562 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 logging 2013-04-03 10:41:01.817339 mon.0 10.252.24.25:6789/0 126 : [INF] pgmap v29060: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1764B/s wr, 0op/s -12884> 2013-04-03 10:41:01.842456 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113603 ==== paxos(lease_ack lc 427405 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1329990796 0 0) 0x29fda00 con 0x208a840 -12883> 2013-04-03 10:41:01.842505 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos -12882> 2013-04-03 10:41:01.842523 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394059 update_from_paxos version 394059 summary v 394059 -12881> 2013-04-03 10:41:01.842538 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12880> 2013-04-03 10:41:02.817031 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394059 encode_pending v394060 -12879> 2013-04-03 10:41:02.817070 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427405) queue_proposal bl 443 bytes; ctx = 0x2c3d690 -12878> 2013-04-03 10:41:02.817077 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427405) propose_queued 427406 443 bytes -12877> 2013-04-03 10:41:02.818177 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427405 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6e500 -12876> 2013-04-03 10:41:02.865511 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113604 ==== paxos(accept lc 427405 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (21716862 0 0) 0x2c6e500 con 0x208a840 -12875> 2013-04-03 10:41:02.866235 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427406 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fda00 -12874> 2013-04-03 10:41:02.866266 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427406 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2c6e000 -12873> 2013-04-03 10:41:02.866303 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos -12872> 2013-04-03 10:41:02.866329 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos version 394060 summary v 394059 -12871> 2013-04-03 10:41:02.866344 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos latest full 393798 -12870> 2013-04-03 10:41:02.866362 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394060 update_from_paxos applying incremental log 394060 2013-04-03 10:41:01.817339 mon.0 10.252.24.25:6789/0 126 : [INF] pgmap v29060: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1764B/s wr, 0op/s -12869> 2013-04-03 10:41:02.866411 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 check_subs -12868> 2013-04-03 10:41:02.866442 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 create_pending v 394061 -12867> 2013-04-03 10:41:02.866464 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394060 _updated_log for mon.0 10.252.24.25:6789/0 -12866> 2013-04-03 10:41:02.866477 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 126) v1 -- ?+0 0x28a58c0 con 0x1bac160 -12865> 2013-04-03 10:41:02.866529 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos -12864> 2013-04-03 10:41:02.866545 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos version 394060 summary v 394060 -12863> 2013-04-03 10:41:02.866563 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12862> 2013-04-03 10:41:02.866584 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 126) v1 ==== 0+0+0 (0 0 0) 0x28a58c0 con 0x1bac160 -12861> 2013-04-03 10:41:02.866592 7fcdffcdb700 10 handle_log_ack log(last 126) v1 -12860> 2013-04-03 10:41:02.866594 7fcdffcdb700 10 logged 2013-04-03 10:41:01.817339 mon.0 10.252.24.25:6789/0 126 : [INF] pgmap v29060: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1764B/s wr, 0op/s -12859> 2013-04-03 10:41:02.891569 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113605 ==== paxos(lease_ack lc 427406 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1833149166 0 0) 0x2c6e000 con 0x208a840 -12858> 2013-04-03 10:41:02.891606 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos -12857> 2013-04-03 10:41:02.891623 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos version 394060 summary v 394060 -12856> 2013-04-03 10:41:02.891638 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12855> 2013-04-03 10:41:03.241252 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113606 ==== forward(pg_stats(1 pgs tid 17 v 605) v1) to leader v1 ==== 757+0+0 (1508014862 0 0) 0x20e3840 con 0x208a840 -12854> 2013-04-03 10:41:03.241280 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -12853> 2013-04-03 10:41:03.241291 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -12852> 2013-04-03 10:41:03.241341 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 preprocess_query pg_stats(1 pgs tid 17 v 605) v1 from osd.2 10.252.24.30:6801/5656 -12851> 2013-04-03 10:41:03.241365 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 prepare_update pg_stats(1 pgs tid 17 v 605) v1 from osd.2 10.252.24.30:6801/5656 -12850> 2013-04-03 10:41:03.241372 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 prepare_pg_stats pg_stats(1 pgs tid 17 v 605) v1 from osd.2 -12849> 2013-04-03 10:41:03.241379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29060 got osd.2 osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -12848> 2013-04-03 10:41:03.354195 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29060 check_down_pgs -12847> 2013-04-03 10:41:03.354224 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29060 v29060: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1764B/s wr, 0op/s -12846> 2013-04-03 10:41:03.354254 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -12845> 2013-04-03 10:41:03.354279 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -12844> 2013-04-03 10:41:03.354315 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -12843> 2013-04-03 10:41:03.354346 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos -12842> 2013-04-03 10:41:03.354364 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos version 394060 summary v 394060 -12841> 2013-04-03 10:41:03.354373 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394060 log -12840> 2013-04-03 10:41:03.354390 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12839> 2013-04-03 10:41:03.354409 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -12838> 2013-04-03 10:41:03.588627 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 113 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140399 v187) v2 ==== 289+0+0 (749190588 0 0) 0x273adc0 con 0x2118420 -12837> 2013-04-03 10:41:03.588674 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140399 v187) v2 from mds.? 10.252.58.31:6800/10908 -12836> 2013-04-03 10:41:03.588685 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140399 v187) v2 -- ?+0 0x20e3840 con 0x2118420 -12835> 2013-04-03 10:41:03.865677 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29060 encode_pending v 29061 -12834> 2013-04-03 10:41:03.865716 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427406) queue_proposal bl 654 bytes; ctx = 0x2c3d550 -12833> 2013-04-03 10:41:03.865725 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427406) propose_queued 427407 654 bytes -12832> 2013-04-03 10:41:03.867265 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427406 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6e280 -12831> 2013-04-03 10:41:03.895022 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113607 ==== paxos(accept lc 427406 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2357417474 0 0) 0x2c6e280 con 0x208a840 -12830> 2013-04-03 10:41:03.895332 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427407 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6e000 -12829> 2013-04-03 10:41:03.895365 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427407 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2c6e500 -12828> 2013-04-03 10:41:03.895436 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29060 update_from_paxos applying incremental 29061 -12827> 2013-04-03 10:41:03.895455 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 v29061: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s -12826> 2013-04-03 10:41:03.895473 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 send_pg_creates to 0 pgs -12825> 2013-04-03 10:41:03.895475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 update_logger -12824> 2013-04-03 10:41:03.895485 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 create_pending v 29062 -12823> 2013-04-03 10:41:03.895496 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29061 _updated_stats for osd.2 10.252.24.30:6801/5656 -12822> 2013-04-03 10:41:03.895503 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 17) v1 tid 17609) v2 -- ?+0 0x24f8d80 con 0x208a840 -12821> 2013-04-03 10:41:03.895515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 check_osd_map already seen 605 >= 605 -12820> 2013-04-03 10:41:03.895519 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 update_logger -12819> 2013-04-03 10:41:03.895529 7fcdffcdb700 0 log [INF] : pgmap v29061: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s -12818> 2013-04-03 10:41:03.895530 7fcdffcdb700 10 send_log to self -12817> 2013-04-03 10:41:03.895537 7fcdffcdb700 10 log_queue is 1 last_log 127 sent 126 num 1 unsent 1 sending 1 -12816> 2013-04-03 10:41:03.895539 7fcdffcdb700 10 will send 2013-04-03 10:41:03.895530 mon.0 10.252.24.25:6789/0 127 : [INF] pgmap v29061: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s -12815> 2013-04-03 10:41:03.895547 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75400 -12814> 2013-04-03 10:41:03.895595 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos -12813> 2013-04-03 10:41:03.895613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos version 394060 summary v 394060 -12812> 2013-04-03 10:41:03.895630 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12811> 2013-04-03 10:41:03.895650 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75400 con 0x1bac160 -12810> 2013-04-03 10:41:03.895672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos -12809> 2013-04-03 10:41:03.895687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos version 394060 summary v 394060 -12808> 2013-04-03 10:41:03.895696 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12807> 2013-04-03 10:41:03.895707 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 preprocess_log log(1 entries) v1 from mon.0 -12806> 2013-04-03 10:41:03.895731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12805> 2013-04-03 10:41:03.895744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 prepare_log log(1 entries) v1 from mon.0 -12804> 2013-04-03 10:41:03.895754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 logging 2013-04-03 10:41:03.895530 mon.0 10.252.24.25:6789/0 127 : [INF] pgmap v29061: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s -12803> 2013-04-03 10:41:03.907208 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113608 ==== paxos(lease_ack lc 427407 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1873355821 0 0) 0x2c6ec80 con 0x208a840 -12802> 2013-04-03 10:41:03.907247 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos -12801> 2013-04-03 10:41:03.907263 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394060 update_from_paxos version 394060 summary v 394060 -12800> 2013-04-03 10:41:03.907280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12799> 2013-04-03 10:41:04.895222 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394060 encode_pending v394061 -12798> 2013-04-03 10:41:04.895249 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427407) queue_proposal bl 443 bytes; ctx = 0x2c3d530 -12797> 2013-04-03 10:41:04.895256 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427407) propose_queued 427408 443 bytes -12796> 2013-04-03 10:41:04.896355 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427407 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d780 -12795> 2013-04-03 10:41:04.932331 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113609 ==== paxos(accept lc 427407 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3438067716 0 0) 0x2787000 con 0x208a840 -12794> 2013-04-03 10:41:04.932648 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427408 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6ec80 -12793> 2013-04-03 10:41:04.932679 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427408 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2c6e280 -12792> 2013-04-03 10:41:04.932718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos -12791> 2013-04-03 10:41:04.932744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394060 -12790> 2013-04-03 10:41:04.932760 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos latest full 393798 -12789> 2013-04-03 10:41:04.932779 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394061 update_from_paxos applying incremental log 394061 2013-04-03 10:41:03.895530 mon.0 10.252.24.25:6789/0 127 : [INF] pgmap v29061: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s -12788> 2013-04-03 10:41:04.932823 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 check_subs -12787> 2013-04-03 10:41:04.932854 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 create_pending v 394062 -12786> 2013-04-03 10:41:04.932876 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394061 _updated_log for mon.0 10.252.24.25:6789/0 -12785> 2013-04-03 10:41:04.932889 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 127) v1 -- ?+0 0x28a5e00 con 0x1bac160 -12784> 2013-04-03 10:41:04.932940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos -12783> 2013-04-03 10:41:04.932955 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 -12782> 2013-04-03 10:41:04.932973 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12781> 2013-04-03 10:41:04.932996 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 127) v1 ==== 0+0+0 (0 0 0) 0x28a5e00 con 0x1bac160 -12780> 2013-04-03 10:41:04.933004 7fcdffcdb700 10 handle_log_ack log(last 127) v1 -12779> 2013-04-03 10:41:04.933007 7fcdffcdb700 10 logged 2013-04-03 10:41:03.895530 mon.0 10.252.24.25:6789/0 127 : [INF] pgmap v29061: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s -12778> 2013-04-03 10:41:04.958374 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113610 ==== paxos(lease_ack lc 427408 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (274397656 0 0) 0x2c6e280 con 0x208a840 -12777> 2013-04-03 10:41:04.958416 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos -12776> 2013-04-03 10:41:04.958432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 -12775> 2013-04-03 10:41:04.958450 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12774> 2013-04-03 10:41:04.958703 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113611 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 114 v187) v2) to leader v1 ==== 560+0+0 (2298787984 0 0) 0x20e4080 con 0x208a840 -12773> 2013-04-03 10:41:04.958715 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -12772> 2013-04-03 10:41:04.958724 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e3dc0 from 10.252.24.88:6789/0 -12771> 2013-04-03 10:41:04.958751 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 114 v187) v2 from mds.0 10.252.24.25:6802/3144 -12770> 2013-04-03 10:41:04.958762 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 114 v187) v2 tid 17610) v2 -- ?+0 0x24f8b40 con 0x208a840 -12769> 2013-04-03 10:41:05.387589 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 113 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21770 v187) v2 ==== 289+0+0 (4278303974 0 0) 0x2a52b80 con 0x200d420 -12768> 2013-04-03 10:41:05.387666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21770 v187) v2 from mds.? 10.252.24.88:6800/17213 -12767> 2013-04-03 10:41:05.387679 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21770 v187) v2 -- ?+0 0x20e4080 con 0x200d420 -12766> 2013-04-03 10:41:07.588754 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 114 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140400 v187) v2 ==== 289+0+0 (3875804900 0 0) 0x273a000 con 0x2118420 -12765> 2013-04-03 10:41:07.588829 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140400 v187) v2 from mds.? 10.252.58.31:6800/10908 -12764> 2013-04-03 10:41:07.588842 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140400 v187) v2 -- ?+0 0x2a52b80 con 0x2118420 -12763> 2013-04-03 10:41:07.932753 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427408 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2c4d500 -12762> 2013-04-03 10:41:07.933505 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113612 ==== paxos(lease_ack lc 427408 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3357587538 0 0) 0x2c4d500 con 0x208a840 -12761> 2013-04-03 10:41:07.933553 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos -12760> 2013-04-03 10:41:07.933573 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 -12759> 2013-04-03 10:41:07.933590 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12758> 2013-04-03 10:41:08.354557 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29061 check_down_pgs -12757> 2013-04-03 10:41:08.354586 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29061 v29061: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s -12756> 2013-04-03 10:41:08.354614 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -12755> 2013-04-03 10:41:08.354638 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -12754> 2013-04-03 10:41:08.354675 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -12753> 2013-04-03 10:41:08.354706 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos -12752> 2013-04-03 10:41:08.354724 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 -12751> 2013-04-03 10:41:08.354734 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394061 log -12750> 2013-04-03 10:41:08.354751 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12749> 2013-04-03 10:41:08.354768 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -12748> 2013-04-03 10:41:08.934983 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113613 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 115 v187) v2) to leader v1 ==== 560+0+0 (2152556179 0 0) 0x273adc0 con 0x208a840 -12747> 2013-04-03 10:41:08.934998 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -12746> 2013-04-03 10:41:08.935007 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4600 from 10.252.24.88:6789/0 -12745> 2013-04-03 10:41:08.935050 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 115 v187) v2 from mds.0 10.252.24.25:6802/3144 -12744> 2013-04-03 10:41:08.935061 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 115 v187) v2 tid 17611) v2 -- ?+0 0x24f8d80 con 0x208a840 -12743> 2013-04-03 10:41:09.387706 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 114 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21771 v187) v2 ==== 289+0+0 (3698262939 0 0) 0x2996340 con 0x200d420 -12742> 2013-04-03 10:41:09.387784 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21771 v187) v2 from mds.? 10.252.24.88:6800/17213 -12741> 2013-04-03 10:41:09.387797 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21771 v187) v2 -- ?+0 0x273adc0 con 0x200d420 -12740> 2013-04-03 10:41:10.932823 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427408 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2c4d280 -12739> 2013-04-03 10:41:10.933572 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113614 ==== paxos(lease_ack lc 427408 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3712784023 0 0) 0x2c4d280 con 0x208a840 -12738> 2013-04-03 10:41:10.933640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos -12737> 2013-04-03 10:41:10.933660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 -12736> 2013-04-03 10:41:10.933677 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12735> 2013-04-03 10:41:11.588870 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 115 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140401 v187) v2 ==== 289+0+0 (3295324569 0 0) 0x273b8c0 con 0x2118420 -12734> 2013-04-03 10:41:11.588952 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140401 v187) v2 from mds.? 10.252.58.31:6800/10908 -12733> 2013-04-03 10:41:11.588965 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140401 v187) v2 -- ?+0 0x2996340 con 0x2118420 -12732> 2013-04-03 10:41:11.600038 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113615 ==== forward(pg_stats(1 pgs tid 38 v 605) v1) to leader v1 ==== 753+0+0 (3513165718 0 0) 0x273a000 con 0x208a840 -12731> 2013-04-03 10:41:11.600051 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -12730> 2013-04-03 10:41:11.600059 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -12729> 2013-04-03 10:41:11.600092 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 preprocess_query pg_stats(1 pgs tid 38 v 605) v1 from osd.1 10.252.24.25:6805/3572 -12728> 2013-04-03 10:41:11.600115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 prepare_update pg_stats(1 pgs tid 38 v 605) v1 from osd.1 10.252.24.25:6805/3572 -12727> 2013-04-03 10:41:11.600121 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 prepare_pg_stats pg_stats(1 pgs tid 38 v 605) v1 from osd.1 -12726> 2013-04-03 10:41:11.600127 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29061 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -12725> 2013-04-03 10:41:11.650313 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29061 encode_pending v 29062 -12724> 2013-04-03 10:41:11.650352 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427408) queue_proposal bl 650 bytes; ctx = 0x2c3d510 -12723> 2013-04-03 10:41:11.650362 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427408) propose_queued 427409 650 bytes -12722> 2013-04-03 10:41:11.651325 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427408 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4a00 -12721> 2013-04-03 10:41:11.683277 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113616 ==== paxos(accept lc 427408 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (946312014 0 0) 0x29b4a00 con 0x208a840 -12720> 2013-04-03 10:41:11.683572 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427409 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d280 -12719> 2013-04-03 10:41:11.683603 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427409 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2c4d500 -12718> 2013-04-03 10:41:11.683672 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29061 update_from_paxos applying incremental 29062 -12717> 2013-04-03 10:41:11.683688 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 v29062: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -12716> 2013-04-03 10:41:11.683706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 send_pg_creates to 0 pgs -12715> 2013-04-03 10:41:11.683710 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 update_logger -12714> 2013-04-03 10:41:11.683720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 create_pending v 29063 -12713> 2013-04-03 10:41:11.683728 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29062 _updated_stats for osd.1 10.252.24.25:6805/3572 -12712> 2013-04-03 10:41:11.683735 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 38) v1 tid 17612) v2 -- ?+0 0x24f8b40 con 0x208a840 -12711> 2013-04-03 10:41:11.683749 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 check_osd_map already seen 605 >= 605 -12710> 2013-04-03 10:41:11.683752 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 update_logger -12709> 2013-04-03 10:41:11.683762 7fcdffcdb700 0 log [INF] : pgmap v29062: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -12708> 2013-04-03 10:41:11.683764 7fcdffcdb700 10 send_log to self -12707> 2013-04-03 10:41:11.683765 7fcdffcdb700 10 log_queue is 1 last_log 128 sent 127 num 1 unsent 1 sending 1 -12706> 2013-04-03 10:41:11.683766 7fcdffcdb700 10 will send 2013-04-03 10:41:11.683763 mon.0 10.252.24.25:6789/0 128 : [INF] pgmap v29062: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -12705> 2013-04-03 10:41:11.683774 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75400 -12704> 2013-04-03 10:41:11.683819 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos -12703> 2013-04-03 10:41:11.683839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 -12702> 2013-04-03 10:41:11.683859 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12701> 2013-04-03 10:41:11.683879 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75400 con 0x1bac160 -12700> 2013-04-03 10:41:11.683901 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos -12699> 2013-04-03 10:41:11.683916 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 -12698> 2013-04-03 10:41:11.683924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12697> 2013-04-03 10:41:11.683933 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 preprocess_log log(1 entries) v1 from mon.0 -12696> 2013-04-03 10:41:11.683956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12695> 2013-04-03 10:41:11.683968 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 prepare_log log(1 entries) v1 from mon.0 -12694> 2013-04-03 10:41:11.683977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 logging 2013-04-03 10:41:11.683763 mon.0 10.252.24.25:6789/0 128 : [INF] pgmap v29062: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -12693> 2013-04-03 10:41:11.695698 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113617 ==== paxos(lease_ack lc 427409 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3120761857 0 0) 0x2c4d500 con 0x208a840 -12692> 2013-04-03 10:41:11.695744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos -12691> 2013-04-03 10:41:11.695762 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394061 update_from_paxos version 394061 summary v 394061 -12690> 2013-04-03 10:41:11.695777 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12689> 2013-04-03 10:41:11.765539 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113618 ==== forward(pg_stats(1 pgs tid 30451 v 605) v1) to leader v1 ==== 757+0+0 (2645960447 0 0) 0x20e48c0 con 0x208a840 -12688> 2013-04-03 10:41:11.765552 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -12687> 2013-04-03 10:41:11.765560 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -12686> 2013-04-03 10:41:11.765594 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 preprocess_query pg_stats(1 pgs tid 30451 v 605) v1 from osd.3 10.252.24.88:6801/17538 -12685> 2013-04-03 10:41:11.765615 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 prepare_update pg_stats(1 pgs tid 30451 v 605) v1 from osd.3 10.252.24.88:6801/17538 -12684> 2013-04-03 10:41:11.765621 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 prepare_pg_stats pg_stats(1 pgs tid 30451 v 605) v1 from osd.3 -12683> 2013-04-03 10:41:11.765627 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29062 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -12682> 2013-04-03 10:41:12.683475 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29062 encode_pending v 29063 -12681> 2013-04-03 10:41:12.683517 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427409) queue_proposal bl 654 bytes; ctx = 0x2c3d670 -12680> 2013-04-03 10:41:12.683527 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427409) propose_queued 427410 654 bytes -12679> 2013-04-03 10:41:12.684823 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427409 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2d26a00 -12678> 2013-04-03 10:41:12.684924 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394061 encode_pending v394062 -12677> 2013-04-03 10:41:12.684950 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427382..427409) queue_proposal bl 443 bytes; ctx = 0x2c3d680 -12676> 2013-04-03 10:41:12.684957 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427382..427409) propose_new_value not active; proposal queued -12675> 2013-04-03 10:41:12.837335 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113619 ==== paxos(accept lc 427409 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2859538342 0 0) 0x2d26a00 con 0x208a840 -12674> 2013-04-03 10:41:12.837683 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427410 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d500 -12673> 2013-04-03 10:41:12.837713 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427410 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29b4a00 -12672> 2013-04-03 10:41:12.837791 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29062 update_from_paxos applying incremental 29063 -12671> 2013-04-03 10:41:12.837817 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 v29063: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -12670> 2013-04-03 10:41:12.837838 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 send_pg_creates to 0 pgs -12669> 2013-04-03 10:41:12.837842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 update_logger -12668> 2013-04-03 10:41:12.837852 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 create_pending v 29064 -12667> 2013-04-03 10:41:12.837861 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29063 _updated_stats for osd.3 10.252.24.88:6801/17538 -12666> 2013-04-03 10:41:12.837868 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30451) v1 tid 17613) v2 -- ?+0 0x24f8d80 con 0x208a840 -12665> 2013-04-03 10:41:12.837878 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 check_osd_map already seen 605 >= 605 -12664> 2013-04-03 10:41:12.837881 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 update_logger -12663> 2013-04-03 10:41:12.837891 7fcdffcdb700 0 log [INF] : pgmap v29063: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -12662> 2013-04-03 10:41:12.837893 7fcdffcdb700 10 send_log to self -12661> 2013-04-03 10:41:12.837893 7fcdffcdb700 10 log_queue is 2 last_log 129 sent 128 num 2 unsent 1 sending 1 -12660> 2013-04-03 10:41:12.837895 7fcdffcdb700 10 will send 2013-04-03 10:41:11.683763 mon.0 10.252.24.25:6789/0 128 : [INF] pgmap v29062: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -12659> 2013-04-03 10:41:12.837903 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75e00 -12658> 2013-04-03 10:41:12.837929 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427410) propose_queued 427411 443 bytes -12657> 2013-04-03 10:41:12.838145 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427410 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6e280 -12656> 2013-04-03 10:41:12.838178 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75e00 con 0x1bac160 -12655> 2013-04-03 10:41:12.863433 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113620 ==== paxos(lease_ack lc 427410 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2457710458 0 0) 0x2c6e280 con 0x208a840 -12654> 2013-04-03 10:41:12.876932 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113621 ==== paxos(accept lc 427410 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (323509224 0 0) 0x29b4a00 con 0x208a840 -12653> 2013-04-03 10:41:12.877214 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427411 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c6e280 -12652> 2013-04-03 10:41:12.877248 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427411 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2d26a00 -12651> 2013-04-03 10:41:12.877282 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos -12650> 2013-04-03 10:41:12.877306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394061 -12649> 2013-04-03 10:41:12.877320 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos latest full 393798 -12648> 2013-04-03 10:41:12.877339 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394062 update_from_paxos applying incremental log 394062 2013-04-03 10:41:11.683763 mon.0 10.252.24.25:6789/0 128 : [INF] pgmap v29062: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -12647> 2013-04-03 10:41:12.877384 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 check_subs -12646> 2013-04-03 10:41:12.877415 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 create_pending v 394063 -12645> 2013-04-03 10:41:12.877437 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394062 _updated_log for mon.0 10.252.24.25:6789/0 -12644> 2013-04-03 10:41:12.877449 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 128) v1 -- ?+0 0x28a5c40 con 0x1bac160 -12643> 2013-04-03 10:41:12.877471 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos -12642> 2013-04-03 10:41:12.877486 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 -12641> 2013-04-03 10:41:12.877495 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12640> 2013-04-03 10:41:12.877506 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 preprocess_log log(1 entries) v1 from mon.0 -12639> 2013-04-03 10:41:12.877524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 nothing new -12638> 2013-04-03 10:41:12.877573 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos -12637> 2013-04-03 10:41:12.877588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 -12636> 2013-04-03 10:41:12.877606 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12635> 2013-04-03 10:41:12.877627 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 128) v1 ==== 0+0+0 (0 0 0) 0x28a5c40 con 0x1bac160 -12634> 2013-04-03 10:41:12.877635 7fcdffcdb700 10 handle_log_ack log(last 128) v1 -12633> 2013-04-03 10:41:12.877636 7fcdffcdb700 10 logged 2013-04-03 10:41:11.683763 mon.0 10.252.24.25:6789/0 128 : [INF] pgmap v29062: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -12632> 2013-04-03 10:41:12.889323 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113622 ==== paxos(lease_ack lc 427411 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2411247180 0 0) 0x2d26a00 con 0x208a840 -12631> 2013-04-03 10:41:12.889357 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos -12630> 2013-04-03 10:41:12.889374 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 -12629> 2013-04-03 10:41:12.889389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12628> 2013-04-03 10:41:12.935052 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113623 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 116 v187) v2) to leader v1 ==== 560+0+0 (647556862 0 0) 0x20e4b80 con 0x208a840 -12627> 2013-04-03 10:41:12.935066 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -12626> 2013-04-03 10:41:12.935074 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e3b00 from 10.252.24.88:6789/0 -12625> 2013-04-03 10:41:12.935105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 116 v187) v2 from mds.0 10.252.24.25:6802/3144 -12624> 2013-04-03 10:41:12.935115 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 116 v187) v2 tid 17614) v2 -- ?+0 0x24f8b40 con 0x208a840 -12623> 2013-04-03 10:41:13.354912 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29063 check_down_pgs -12622> 2013-04-03 10:41:13.354947 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29063 v29063: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -12621> 2013-04-03 10:41:13.355256 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -12620> 2013-04-03 10:41:13.355280 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -12619> 2013-04-03 10:41:13.355318 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -12618> 2013-04-03 10:41:13.355349 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos -12617> 2013-04-03 10:41:13.355367 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 -12616> 2013-04-03 10:41:13.355376 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394062 log -12615> 2013-04-03 10:41:13.355393 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12614> 2013-04-03 10:41:13.355412 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -12613> 2013-04-03 10:41:13.387811 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 115 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21772 v187) v2 ==== 289+0+0 (879116264 0 0) 0x2ce2340 con 0x200d420 -12612> 2013-04-03 10:41:13.387856 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21772 v187) v2 from mds.? 10.252.24.88:6800/17213 -12611> 2013-04-03 10:41:13.387867 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21772 v187) v2 -- ?+0 0x20e4b80 con 0x200d420 -12610> 2013-04-03 10:41:15.588980 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 116 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140402 v187) v2 ==== 289+0+0 (2715379742 0 0) 0x273b600 con 0x2118420 -12609> 2013-04-03 10:41:15.589056 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140402 v187) v2 from mds.? 10.252.58.31:6800/10908 -12608> 2013-04-03 10:41:15.589069 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140402 v187) v2 -- ?+0 0x2ce2340 con 0x2118420 -12607> 2013-04-03 10:41:15.877323 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427411 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2d26780 -12606> 2013-04-03 10:41:15.878109 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113624 ==== paxos(lease_ack lc 427411 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2938182468 0 0) 0x2d26780 con 0x208a840 -12605> 2013-04-03 10:41:15.878155 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos -12604> 2013-04-03 10:41:15.878175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 -12603> 2013-04-03 10:41:15.878192 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12602> 2013-04-03 10:41:16.935208 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113625 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 117 v187) v2) to leader v1 ==== 560+0+0 (802238717 0 0) 0x20e48c0 con 0x208a840 -12601> 2013-04-03 10:41:16.935228 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -12600> 2013-04-03 10:41:16.935238 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2145b00 from 10.252.24.88:6789/0 -12599> 2013-04-03 10:41:16.935291 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 117 v187) v2 from mds.0 10.252.24.25:6802/3144 -12598> 2013-04-03 10:41:16.935303 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 117 v187) v2 tid 17615) v2 -- ?+0 0x24f8d80 con 0x208a840 -12597> 2013-04-03 10:41:17.387899 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 116 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21773 v187) v2 ==== 289+0+0 (386433173 0 0) 0x2ce2080 con 0x200d420 -12596> 2013-04-03 10:41:17.387977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21773 v187) v2 from mds.? 10.252.24.88:6800/17213 -12595> 2013-04-03 10:41:17.387990 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21773 v187) v2 -- ?+0 0x20e48c0 con 0x200d420 -12594> 2013-04-03 10:41:17.417976 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 117 ==== mon_subscribe({mdsmap=188+,monmap=6+}) v2 ==== 42+0+0 (435898880 0 0) 0x2e8b380 con 0x200d420 -12593> 2013-04-03 10:41:17.417989 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 -12592> 2013-04-03 10:41:17.417997 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -12591> 2013-04-03 10:41:17.418000 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/17213 -- mon_subscribe_ack(300s) v1 -- ?+0 0x28a5c40 -12590> 2013-04-03 10:41:18.243211 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113626 ==== forward(pg_stats(1 pgs tid 18 v 605) v1) to leader v1 ==== 757+0+0 (2515128539 0 0) 0x273b600 con 0x208a840 -12589> 2013-04-03 10:41:18.243232 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -12588> 2013-04-03 10:41:18.243240 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -12587> 2013-04-03 10:41:18.243277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 preprocess_query pg_stats(1 pgs tid 18 v 605) v1 from osd.2 10.252.24.30:6801/5656 -12586> 2013-04-03 10:41:18.243299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 prepare_update pg_stats(1 pgs tid 18 v 605) v1 from osd.2 10.252.24.30:6801/5656 -12585> 2013-04-03 10:41:18.243303 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 prepare_pg_stats pg_stats(1 pgs tid 18 v 605) v1 from osd.2 -12584> 2013-04-03 10:41:18.243309 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29063 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1175 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -12583> 2013-04-03 10:41:18.293500 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29063 encode_pending v 29064 -12582> 2013-04-03 10:41:18.293542 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427411) queue_proposal bl 654 bytes; ctx = 0x2c3d560 -12581> 2013-04-03 10:41:18.293557 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427411) propose_queued 427412 654 bytes -12580> 2013-04-03 10:41:18.294650 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427411 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4c80 -12579> 2013-04-03 10:41:18.319643 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113627 ==== paxos(accept lc 427411 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1763998887 0 0) 0x20f4c80 con 0x208a840 -12578> 2013-04-03 10:41:18.319929 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427412 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2d26780 -12577> 2013-04-03 10:41:18.319959 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427412 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2d26a00 -12576> 2013-04-03 10:41:18.320029 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29063 update_from_paxos applying incremental 29064 -12575> 2013-04-03 10:41:18.320049 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 v29064: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -12574> 2013-04-03 10:41:18.320068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 send_pg_creates to 0 pgs -12573> 2013-04-03 10:41:18.320072 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 update_logger -12572> 2013-04-03 10:41:18.320082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 create_pending v 29065 -12571> 2013-04-03 10:41:18.320091 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29064 _updated_stats for osd.2 10.252.24.30:6801/5656 -12570> 2013-04-03 10:41:18.320098 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 18) v1 tid 17616) v2 -- ?+0 0x24f8b40 con 0x208a840 -12569> 2013-04-03 10:41:18.320110 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 check_osd_map already seen 605 >= 605 -12568> 2013-04-03 10:41:18.320115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 update_logger -12567> 2013-04-03 10:41:18.320126 7fcdffcdb700 0 log [INF] : pgmap v29064: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -12566> 2013-04-03 10:41:18.320135 7fcdffcdb700 10 send_log to self -12565> 2013-04-03 10:41:18.320136 7fcdffcdb700 10 log_queue is 2 last_log 130 sent 128 num 2 unsent 2 sending 2 -12564> 2013-04-03 10:41:18.320138 7fcdffcdb700 10 will send 2013-04-03 10:41:12.837892 mon.0 10.252.24.25:6789/0 129 : [INF] pgmap v29063: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -12563> 2013-04-03 10:41:18.320145 7fcdffcdb700 10 will send 2013-04-03 10:41:18.320134 mon.0 10.252.24.25:6789/0 130 : [INF] pgmap v29064: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -12562> 2013-04-03 10:41:18.320151 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b75e00 -12561> 2013-04-03 10:41:18.320203 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos -12560> 2013-04-03 10:41:18.320222 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 -12559> 2013-04-03 10:41:18.320241 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12558> 2013-04-03 10:41:18.320262 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75e00 con 0x1bac160 -12557> 2013-04-03 10:41:18.320284 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos -12556> 2013-04-03 10:41:18.320299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 -12555> 2013-04-03 10:41:18.320308 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -12554> 2013-04-03 10:41:18.320319 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 preprocess_log log(2 entries) v1 from mon.0 -12553> 2013-04-03 10:41:18.320351 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -12552> 2013-04-03 10:41:18.320363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 prepare_log log(2 entries) v1 from mon.0 -12551> 2013-04-03 10:41:18.320374 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 logging 2013-04-03 10:41:12.837892 mon.0 10.252.24.25:6789/0 129 : [INF] pgmap v29063: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -12550> 2013-04-03 10:41:18.320396 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 logging 2013-04-03 10:41:18.320134 mon.0 10.252.24.25:6789/0 130 : [INF] pgmap v29064: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -12549> 2013-04-03 10:41:18.345685 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113628 ==== paxos(lease_ack lc 427412 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (4153763800 0 0) 0x2d26a00 con 0x208a840 -12548> 2013-04-03 10:41:18.345722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos -12547> 2013-04-03 10:41:18.345740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 -12546> 2013-04-03 10:41:18.345756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12545> 2013-04-03 10:41:18.355526 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29064 check_down_pgs -12544> 2013-04-03 10:41:18.355561 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29064 v29064: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -12543> 2013-04-03 10:41:18.355584 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -12542> 2013-04-03 10:41:18.355606 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -12541> 2013-04-03 10:41:18.355643 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -12540> 2013-04-03 10:41:18.355673 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos -12539> 2013-04-03 10:41:18.355690 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394062 update_from_paxos version 394062 summary v 394062 -12538> 2013-04-03 10:41:18.355699 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394062 log -12537> 2013-04-03 10:41:18.355715 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12536> 2013-04-03 10:41:18.355732 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -12535> 2013-04-03 10:41:19.319848 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394062 encode_pending v394063 -12534> 2013-04-03 10:41:19.319877 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427412) queue_proposal bl 725 bytes; ctx = 0x2c3d660 -12533> 2013-04-03 10:41:19.319884 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427412) propose_queued 427413 725 bytes -12532> 2013-04-03 10:41:19.321104 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427412 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4500 -12531> 2013-04-03 10:41:19.345753 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113629 ==== paxos(accept lc 427412 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (515045606 0 0) 0x20f4500 con 0x208a840 -12530> 2013-04-03 10:41:19.346047 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427413 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2d26a00 -12529> 2013-04-03 10:41:19.346065 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427413 fc 427382 pn 0 opn 0) v3 -- ?+0 0x20f4c80 -12528> 2013-04-03 10:41:19.346108 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos -12527> 2013-04-03 10:41:19.346135 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394062 -12526> 2013-04-03 10:41:19.346149 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos latest full 393798 -12525> 2013-04-03 10:41:19.346168 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394063 update_from_paxos applying incremental log 394063 2013-04-03 10:41:12.837892 mon.0 10.252.24.25:6789/0 129 : [INF] pgmap v29063: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -12524> 2013-04-03 10:41:19.346196 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394063 update_from_paxos applying incremental log 394063 2013-04-03 10:41:18.320134 mon.0 10.252.24.25:6789/0 130 : [INF] pgmap v29064: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -12523> 2013-04-03 10:41:19.346235 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 check_subs -12522> 2013-04-03 10:41:19.346266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 create_pending v 394064 -12521> 2013-04-03 10:41:19.346288 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394063 _updated_log for mon.0 10.252.24.25:6789/0 -12520> 2013-04-03 10:41:19.346301 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 130) v1 -- ?+0 0x28a51c0 con 0x1bac160 -12519> 2013-04-03 10:41:19.346353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos -12518> 2013-04-03 10:41:19.346368 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 -12517> 2013-04-03 10:41:19.346386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12516> 2013-04-03 10:41:19.346409 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 130) v1 ==== 0+0+0 (0 0 0) 0x28a51c0 con 0x1bac160 -12515> 2013-04-03 10:41:19.346417 7fcdffcdb700 10 handle_log_ack log(last 130) v1 -12514> 2013-04-03 10:41:19.346421 7fcdffcdb700 10 logged 2013-04-03 10:41:12.837892 mon.0 10.252.24.25:6789/0 129 : [INF] pgmap v29063: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 580B/s wr, 0op/s -12513> 2013-04-03 10:41:19.346427 7fcdffcdb700 10 logged 2013-04-03 10:41:18.320134 mon.0 10.252.24.25:6789/0 130 : [INF] pgmap v29064: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -12512> 2013-04-03 10:41:19.359460 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113630 ==== paxos(lease_ack lc 427413 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3157679109 0 0) 0x20f4c80 con 0x208a840 -12511> 2013-04-03 10:41:19.359502 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos -12510> 2013-04-03 10:41:19.359519 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 -12509> 2013-04-03 10:41:19.359535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12508> 2013-04-03 10:41:19.589125 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 117 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140403 v187) v2 ==== 289+0+0 (2193086307 0 0) 0x273b080 con 0x2118420 -12507> 2013-04-03 10:41:19.589170 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140403 v187) v2 from mds.? 10.252.58.31:6800/10908 -12506> 2013-04-03 10:41:19.589181 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140403 v187) v2 -- ?+0 0x273b600 con 0x2118420 -12505> 2013-04-03 10:41:20.935397 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113631 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 118 v187) v2) to leader v1 ==== 560+0+0 (4029523301 0 0) 0x2145840 con 0x208a840 -12504> 2013-04-03 10:41:20.935417 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -12503> 2013-04-03 10:41:20.935426 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x21452c0 from 10.252.24.88:6789/0 -12502> 2013-04-03 10:41:20.935479 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 118 v187) v2 from mds.0 10.252.24.25:6802/3144 -12501> 2013-04-03 10:41:20.935491 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 118 v187) v2 tid 17617) v2 -- ?+0 0x24f8240 con 0x208a840 -12500> 2013-04-03 10:41:21.408980 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 118 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21774 v187) v2 ==== 289+0+0 (1924874514 0 0) 0x2ce2b80 con 0x200d420 -12499> 2013-04-03 10:41:21.409058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21774 v187) v2 from mds.? 10.252.24.88:6800/17213 -12498> 2013-04-03 10:41:21.409071 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21774 v187) v2 -- ?+0 0x2145840 con 0x200d420 -12497> 2013-04-03 10:41:22.346140 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427413 fc 427382 pn 0 opn 0) v3 -- ?+0 0x20f4280 -12496> 2013-04-03 10:41:22.346913 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113632 ==== paxos(lease_ack lc 427413 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (941340453 0 0) 0x20f4280 con 0x208a840 -12495> 2013-04-03 10:41:22.346983 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos -12494> 2013-04-03 10:41:22.347005 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 -12493> 2013-04-03 10:41:22.347022 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12492> 2013-04-03 10:41:23.356017 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29064 v29064: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -12491> 2013-04-03 10:41:23.356058 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -12490> 2013-04-03 10:41:23.356084 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -12489> 2013-04-03 10:41:23.356128 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -12488> 2013-04-03 10:41:23.356159 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos -12487> 2013-04-03 10:41:23.356177 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 -12486> 2013-04-03 10:41:23.356186 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394063 log -12485> 2013-04-03 10:41:23.356203 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12484> 2013-04-03 10:41:23.356222 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -12483> 2013-04-03 10:41:23.589234 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 118 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140404 v187) v2 ==== 289+0+0 (1790896912 0 0) 0x2ce28c0 con 0x2118420 -12482> 2013-04-03 10:41:23.589280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140404 v187) v2 from mds.? 10.252.58.31:6800/10908 -12481> 2013-04-03 10:41:23.589291 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140404 v187) v2 -- ?+0 0x2ce2b80 con 0x2118420 -12480> 2013-04-03 10:41:24.935643 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113633 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 119 v187) v2) to leader v1 ==== 560+0+0 (1632430338 0 0) 0x273b080 con 0x208a840 -12479> 2013-04-03 10:41:24.935663 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -12478> 2013-04-03 10:41:24.935672 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2145dc0 from 10.252.24.88:6789/0 -12477> 2013-04-03 10:41:24.935725 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 119 v187) v2 from mds.0 10.252.24.25:6802/3144 -12476> 2013-04-03 10:41:24.935735 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 119 v187) v2 tid 17618) v2 -- ?+0 0x24f8b40 con 0x208a840 -12475> 2013-04-03 10:41:25.346208 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427413 fc 427382 pn 0 opn 0) v3 -- ?+0 0x20f4a00 -12474> 2013-04-03 10:41:25.347103 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113634 ==== paxos(lease_ack lc 427413 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1385866418 0 0) 0x20f4a00 con 0x208a840 -12473> 2013-04-03 10:41:25.347175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos -12472> 2013-04-03 10:41:25.347196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 -12471> 2013-04-03 10:41:25.347213 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12470> 2013-04-03 10:41:25.388112 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 119 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21775 v187) v2 ==== 289+0+0 (1372978799 0 0) 0x2ce1dc0 con 0x200d420 -12469> 2013-04-03 10:41:25.388185 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21775 v187) v2 from mds.? 10.252.24.88:6800/17213 -12468> 2013-04-03 10:41:25.388198 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21775 v187) v2 -- ?+0 0x273b080 con 0x200d420 -12467> 2013-04-03 10:41:25.482938 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 119 ==== mon_subscribe({mdsmap=188+,monmap=6+}) v2 ==== 42+0+0 (435898880 0 0) 0x2e8b8c0 con 0x2118420 -12466> 2013-04-03 10:41:25.482953 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 -12465> 2013-04-03 10:41:25.482964 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -12464> 2013-04-03 10:41:25.482966 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.58.31:6800/10908 -- mon_subscribe_ack(300s) v1 -- ?+0 0x28a51c0 -12463> 2013-04-03 10:41:26.601703 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113635 ==== forward(pg_stats(1 pgs tid 39 v 605) v1) to leader v1 ==== 753+0+0 (3224530509 0 0) 0x2ce28c0 con 0x208a840 -12462> 2013-04-03 10:41:26.601722 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -12461> 2013-04-03 10:41:26.601732 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -12460> 2013-04-03 10:41:26.601782 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 preprocess_query pg_stats(1 pgs tid 39 v 605) v1 from osd.1 10.252.24.25:6805/3572 -12459> 2013-04-03 10:41:26.601806 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 prepare_update pg_stats(1 pgs tid 39 v 605) v1 from osd.1 10.252.24.25:6805/3572 -12458> 2013-04-03 10:41:26.601812 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 prepare_pg_stats pg_stats(1 pgs tid 39 v 605) v1 from osd.1 -12457> 2013-04-03 10:41:26.601818 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29064 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -12456> 2013-04-03 10:41:26.652002 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29064 encode_pending v 29065 -12455> 2013-04-03 10:41:26.652041 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427413) queue_proposal bl 650 bytes; ctx = 0x2c3d780 -12454> 2013-04-03 10:41:26.652050 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427413) propose_queued 427414 650 bytes -12453> 2013-04-03 10:41:26.653167 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427413 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4780 -12452> 2013-04-03 10:41:26.695782 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113636 ==== paxos(accept lc 427413 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1296049980 0 0) 0x20f4780 con 0x208a840 -12451> 2013-04-03 10:41:26.696100 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427414 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4a00 -12450> 2013-04-03 10:41:26.696131 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427414 fc 427382 pn 0 opn 0) v3 -- ?+0 0x20f4280 -12449> 2013-04-03 10:41:26.696200 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29064 update_from_paxos applying incremental 29065 -12448> 2013-04-03 10:41:26.696216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29065 v29065: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -12447> 2013-04-03 10:41:26.696233 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29065 send_pg_creates to 0 pgs -12446> 2013-04-03 10:41:26.696237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29065 update_logger -12445> 2013-04-03 10:41:26.696247 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29065 create_pending v 29066 -12444> 2013-04-03 10:41:26.696256 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29065 _updated_stats for osd.1 10.252.24.25:6805/3572 -12443> 2013-04-03 10:41:26.696263 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 39) v1 tid 17619) v2 -- ?+0 0x24f8240 con 0x208a840 -12442> 2013-04-03 10:41:26.696274 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29065 check_osd_map already seen 605 >= 605 -12441> 2013-04-03 10:41:26.696277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29065 update_logger -12440> 2013-04-03 10:41:26.696288 7fcdffcdb700 0 log [INF] : pgmap v29065: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -12439> 2013-04-03 10:41:26.696289 7fcdffcdb700 10 send_log to self -12438> 2013-04-03 10:41:26.696290 7fcdffcdb700 10 log_queue is 1 last_log 131 sent 130 num 1 unsent 1 sending 1 -12437> 2013-04-03 10:41:26.696292 7fcdffcdb700 10 will send 2013-04-03 10:41:26.696289 mon.0 10.252.24.25:6789/0 131 : [INF] pgmap v29065: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -12436> 2013-04-03 10:41:26.696298 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75e00 -12435> 2013-04-03 10:41:26.696348 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos -12434> 2013-04-03 10:41:26.696366 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 -12433> 2013-04-03 10:41:26.696385 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12432> 2013-04-03 10:41:26.696406 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75e00 con 0x1bac160 -12431> 2013-04-03 10:41:26.696427 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos -12430> 2013-04-03 10:41:26.696442 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 -12429> 2013-04-03 10:41:26.696451 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12428> 2013-04-03 10:41:26.696462 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 preprocess_log log(1 entries) v1 from mon.0 -12427> 2013-04-03 10:41:26.696486 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12426> 2013-04-03 10:41:26.696498 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 prepare_log log(1 entries) v1 from mon.0 -12425> 2013-04-03 10:41:26.696508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 logging 2013-04-03 10:41:26.696289 mon.0 10.252.24.25:6789/0 131 : [INF] pgmap v29065: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -12424> 2013-04-03 10:41:26.721860 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113637 ==== paxos(lease_ack lc 427414 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3262096353 0 0) 0x20f4280 con 0x208a840 -12423> 2013-04-03 10:41:26.721903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos -12422> 2013-04-03 10:41:26.721920 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394063 update_from_paxos version 394063 summary v 394063 -12421> 2013-04-03 10:41:26.721935 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12420> 2013-04-03 10:41:27.589355 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 120 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140405 v187) v2 ==== 289+0+0 (1238389869 0 0) 0x2ce2600 con 0x2118420 -12419> 2013-04-03 10:41:27.589429 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140405 v187) v2 from mds.? 10.252.58.31:6800/10908 -12418> 2013-04-03 10:41:27.589442 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140405 v187) v2 -- ?+0 0x2ce28c0 con 0x2118420 -12417> 2013-04-03 10:41:27.695985 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394063 encode_pending v394064 -12416> 2013-04-03 10:41:27.696012 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427414) queue_proposal bl 443 bytes; ctx = 0x2c3d640 -12415> 2013-04-03 10:41:27.696020 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427414) propose_queued 427415 443 bytes -12414> 2013-04-03 10:41:27.696947 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427414 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787500 -12413> 2013-04-03 10:41:27.880108 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113638 ==== paxos(accept lc 427414 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3796848355 0 0) 0x2787500 con 0x208a840 -12412> 2013-04-03 10:41:27.880384 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427415 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4280 -12411> 2013-04-03 10:41:27.880404 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427415 fc 427382 pn 0 opn 0) v3 -- ?+0 0x20f4780 -12410> 2013-04-03 10:41:27.880445 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos -12409> 2013-04-03 10:41:27.880477 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos version 394064 summary v 394063 -12408> 2013-04-03 10:41:27.880491 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos latest full 393798 -12407> 2013-04-03 10:41:27.880509 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394064 update_from_paxos applying incremental log 394064 2013-04-03 10:41:26.696289 mon.0 10.252.24.25:6789/0 131 : [INF] pgmap v29065: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -12406> 2013-04-03 10:41:27.880555 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 check_subs -12405> 2013-04-03 10:41:27.880587 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 create_pending v 394065 -12404> 2013-04-03 10:41:27.880609 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394064 _updated_log for mon.0 10.252.24.25:6789/0 -12403> 2013-04-03 10:41:27.880621 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 131) v1 -- ?+0 0x2224380 con 0x1bac160 -12402> 2013-04-03 10:41:27.880673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos -12401> 2013-04-03 10:41:27.880689 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos version 394064 summary v 394064 -12400> 2013-04-03 10:41:27.880711 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12399> 2013-04-03 10:41:27.880732 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 131) v1 ==== 0+0+0 (0 0 0) 0x2224380 con 0x1bac160 -12398> 2013-04-03 10:41:27.880740 7fcdffcdb700 10 handle_log_ack log(last 131) v1 -12397> 2013-04-03 10:41:27.880741 7fcdffcdb700 10 logged 2013-04-03 10:41:26.696289 mon.0 10.252.24.25:6789/0 131 : [INF] pgmap v29065: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -12396> 2013-04-03 10:41:27.893835 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113639 ==== paxos(lease_ack lc 427415 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2919634149 0 0) 0x20f4780 con 0x208a840 -12395> 2013-04-03 10:41:27.893909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos -12394> 2013-04-03 10:41:27.893928 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos version 394064 summary v 394064 -12393> 2013-04-03 10:41:27.893945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12392> 2013-04-03 10:41:28.356369 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29065 check_down_pgs -12391> 2013-04-03 10:41:28.356398 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29065 v29065: 240 pgs: 240 active+clean; 254 MB data, 5657 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -12390> 2013-04-03 10:41:28.356426 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -12389> 2013-04-03 10:41:28.356455 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -12388> 2013-04-03 10:41:28.356492 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -12387> 2013-04-03 10:41:28.356522 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos -12386> 2013-04-03 10:41:28.356541 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394064 update_from_paxos version 394064 summary v 394064 -12385> 2013-04-03 10:41:28.356550 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394064 log -12384> 2013-04-03 10:41:28.356567 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -12383> 2013-04-03 10:41:28.356585 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -12382> 2013-04-03 10:41:28.597392 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113640 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (424405211 0 0) 0x2145580 con 0x208a840 -12381> 2013-04-03 10:41:28.597413 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -12380> 2013-04-03 10:41:28.597419 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -12379> 2013-04-03 10:41:28.597428 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 -12378> 2013-04-03 10:41:28.597432 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( start_reply flags( reply-to(mon.2 10.252.24.88:6789/0) ) ) v1 -- ?+0 0x2ce2600 -12377> 2013-04-03 10:41:28.935593 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113641 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 120 v187) v2) to leader v1 ==== 560+0+0 (848424531 0 0) 0x2146080 con 0x208a840 -12376> 2013-04-03 10:41:28.935609 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -12375> 2013-04-03 10:41:28.935618 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x1bae600 from 10.252.24.88:6789/0 -12374> 2013-04-03 10:41:28.935671 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 120 v187) v2 from mds.0 10.252.24.25:6802/3144 -12373> 2013-04-03 10:41:28.935684 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 120 v187) v2 tid 17620) v2 -- ?+0 0x24f8b40 con 0x208a840 -12372> 2013-04-03 10:41:29.388184 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 120 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21776 v187) v2 ==== 289+0+0 (2591446327 0 0) 0x20e4340 con 0x200d420 -12371> 2013-04-03 10:41:29.388263 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21776 v187) v2 from mds.? 10.252.24.88:6800/17213 -12370> 2013-04-03 10:41:29.388278 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21776 v187) v2 -- ?+0 0x2146080 con 0x200d420 -12369> 2013-04-03 10:41:30.777213 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2df2500 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33085/0 -12368> 2013-04-03 10:41:30.777240 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2500 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33085/0) -12367> 2013-04-03 10:41:30.777257 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2500 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -12366> 2013-04-03 10:41:30.777461 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2500 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -12365> 2013-04-03 10:41:30.880479 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427415 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2787c80 -12364> 2013-04-03 10:41:30.881282 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113642 ==== paxos(lease_ack lc 427415 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2836914417 0 0) 0x2787c80 con 0x208a840 -12363> 2013-04-03 10:41:30.881357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos -12362> 2013-04-03 10:41:30.881378 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos version 394064 summary v 394064 -12361> 2013-04-03 10:41:30.881396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12360> 2013-04-03 10:41:31.589496 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 121 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140406 v187) v2 ==== 289+0+0 (744723946 0 0) 0x2ce1000 con 0x2118420 -12359> 2013-04-03 10:41:31.589584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140406 v187) v2 from mds.? 10.252.58.31:6800/10908 -12358> 2013-04-03 10:41:31.589599 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140406 v187) v2 -- ?+0 0x20e4340 con 0x2118420 -12357> 2013-04-03 10:41:32.935773 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113643 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 121 v187) v2) to leader v1 ==== 560+0+0 (1004023888 0 0) 0x2145580 con 0x208a840 -12356> 2013-04-03 10:41:32.935795 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -12355> 2013-04-03 10:41:32.935808 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x1bae340 from 10.252.24.88:6789/0 -12354> 2013-04-03 10:41:32.935860 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 121 v187) v2 from mds.0 10.252.24.25:6802/3144 -12353> 2013-04-03 10:41:32.935871 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 121 v187) v2 tid 17621) v2 -- ?+0 0x24f8240 con 0x208a840 -12352> 2013-04-03 10:41:33.245184 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113644 ==== forward(pg_stats(1 pgs tid 19 v 605) v1) to leader v1 ==== 757+0+0 (2599378194 0 0) 0x2ce1000 con 0x208a840 -12351> 2013-04-03 10:41:33.245200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -12350> 2013-04-03 10:41:33.245208 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -12349> 2013-04-03 10:41:33.245241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29065 preprocess_query pg_stats(1 pgs tid 19 v 605) v1 from osd.2 10.252.24.30:6801/5656 -12348> 2013-04-03 10:41:33.245265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29065 prepare_update pg_stats(1 pgs tid 19 v 605) v1 from osd.2 10.252.24.30:6801/5656 -12347> 2013-04-03 10:41:33.245273 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29065 prepare_pg_stats pg_stats(1 pgs tid 19 v 605) v1 from osd.2 -12346> 2013-04-03 10:41:33.245279 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29065 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -12345> 2013-04-03 10:41:33.295450 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29065 encode_pending v 29066 -12344> 2013-04-03 10:41:33.295488 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427415) queue_proposal bl 654 bytes; ctx = 0x2c3d770 -12343> 2013-04-03 10:41:33.295498 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427415) propose_queued 427416 654 bytes -12342> 2013-04-03 10:41:33.296451 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427415 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787a00 -12341> 2013-04-03 10:41:33.325089 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113645 ==== paxos(accept lc 427415 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1225327676 0 0) 0x2787a00 con 0x208a840 -12340> 2013-04-03 10:41:33.326352 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427416 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787c80 -12339> 2013-04-03 10:41:33.326385 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427416 fc 427382 pn 0 opn 0) v3 -- ?+0 0x20f4780 -12338> 2013-04-03 10:41:33.326455 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29065 update_from_paxos applying incremental 29066 -12337> 2013-04-03 10:41:33.326476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 v29066: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -12336> 2013-04-03 10:41:33.326496 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 send_pg_creates to 0 pgs -12335> 2013-04-03 10:41:33.326500 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 update_logger -12334> 2013-04-03 10:41:33.326511 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 create_pending v 29067 -12333> 2013-04-03 10:41:33.326520 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 _updated_stats for osd.2 10.252.24.30:6801/5656 -12332> 2013-04-03 10:41:33.326528 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 19) v1 tid 17622) v2 -- ?+0 0x24f8b40 con 0x208a840 -12331> 2013-04-03 10:41:33.326538 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 check_osd_map already seen 605 >= 605 -12330> 2013-04-03 10:41:33.326541 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 update_logger -12329> 2013-04-03 10:41:33.326553 7fcdffcdb700 0 log [INF] : pgmap v29066: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -12328> 2013-04-03 10:41:33.326562 7fcdffcdb700 10 send_log to self -12327> 2013-04-03 10:41:33.326564 7fcdffcdb700 10 log_queue is 1 last_log 132 sent 131 num 1 unsent 1 sending 1 -12326> 2013-04-03 10:41:33.326565 7fcdffcdb700 10 will send 2013-04-03 10:41:33.326561 mon.0 10.252.24.25:6789/0 132 : [INF] pgmap v29066: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -12325> 2013-04-03 10:41:33.326573 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75e00 -12324> 2013-04-03 10:41:33.326626 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos -12323> 2013-04-03 10:41:33.326646 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos version 394064 summary v 394064 -12322> 2013-04-03 10:41:33.326665 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12321> 2013-04-03 10:41:33.326687 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75e00 con 0x1bac160 -12320> 2013-04-03 10:41:33.326709 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos -12319> 2013-04-03 10:41:33.326724 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos version 394064 summary v 394064 -12318> 2013-04-03 10:41:33.326733 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12317> 2013-04-03 10:41:33.326746 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 preprocess_log log(1 entries) v1 from mon.0 -12316> 2013-04-03 10:41:33.326771 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12315> 2013-04-03 10:41:33.326784 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 prepare_log log(1 entries) v1 from mon.0 -12314> 2013-04-03 10:41:33.326795 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 logging 2013-04-03 10:41:33.326561 mon.0 10.252.24.25:6789/0 132 : [INF] pgmap v29066: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -12313> 2013-04-03 10:41:33.351161 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113646 ==== paxos(lease_ack lc 427416 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2242743839 0 0) 0x20f4780 con 0x208a840 -12312> 2013-04-03 10:41:33.351205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos -12311> 2013-04-03 10:41:33.351222 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos version 394064 summary v 394064 -12310> 2013-04-03 10:41:33.351238 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12309> 2013-04-03 10:41:33.356688 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 check_down_pgs -12308> 2013-04-03 10:41:33.356718 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 v29066: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -12307> 2013-04-03 10:41:33.356748 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -12306> 2013-04-03 10:41:33.356770 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -12305> 2013-04-03 10:41:33.356807 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -12304> 2013-04-03 10:41:33.356837 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos -12303> 2013-04-03 10:41:33.356854 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 update_from_paxos version 394064 summary v 394064 -12302> 2013-04-03 10:41:33.356865 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 log -12301> 2013-04-03 10:41:33.356881 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12300> 2013-04-03 10:41:33.356897 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -12299> 2013-04-03 10:41:33.388304 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 121 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21777 v187) v2 ==== 289+0+0 (3105417802 0 0) 0x2a84340 con 0x200d420 -12298> 2013-04-03 10:41:33.388354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21777 v187) v2 from mds.? 10.252.24.88:6800/17213 -12297> 2013-04-03 10:41:33.388366 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21777 v187) v2 -- ?+0 0x2ce1000 con 0x200d420 -12296> 2013-04-03 10:41:34.325297 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394064 encode_pending v394065 -12295> 2013-04-03 10:41:34.325335 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427416) queue_proposal bl 443 bytes; ctx = 0x2c3d630 -12294> 2013-04-03 10:41:34.325343 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427416) propose_queued 427417 443 bytes -12293> 2013-04-03 10:41:34.326387 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427416 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787780 -12292> 2013-04-03 10:41:34.354432 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113647 ==== paxos(accept lc 427416 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1277313381 0 0) 0x2787780 con 0x208a840 -12291> 2013-04-03 10:41:34.359218 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427417 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4780 -12290> 2013-04-03 10:41:34.359250 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427417 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2787a00 -12289> 2013-04-03 10:41:34.359286 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos -12288> 2013-04-03 10:41:34.359314 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos version 394065 summary v 394064 -12287> 2013-04-03 10:41:34.359329 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos latest full 393798 -12286> 2013-04-03 10:41:34.359348 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos applying incremental log 394065 2013-04-03 10:41:33.326561 mon.0 10.252.24.25:6789/0 132 : [INF] pgmap v29066: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -12285> 2013-04-03 10:41:34.359392 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 check_subs -12284> 2013-04-03 10:41:34.359424 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 create_pending v 394066 -12283> 2013-04-03 10:41:34.359447 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 _updated_log for mon.0 10.252.24.25:6789/0 -12282> 2013-04-03 10:41:34.359460 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 132) v1 -- ?+0 0x2224000 con 0x1bac160 -12281> 2013-04-03 10:41:34.359512 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos -12280> 2013-04-03 10:41:34.359529 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos version 394065 summary v 394065 -12279> 2013-04-03 10:41:34.359548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12278> 2013-04-03 10:41:34.359572 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 132) v1 ==== 0+0+0 (0 0 0) 0x2224000 con 0x1bac160 -12277> 2013-04-03 10:41:34.359580 7fcdffcdb700 10 handle_log_ack log(last 132) v1 -12276> 2013-04-03 10:41:34.359584 7fcdffcdb700 10 logged 2013-04-03 10:41:33.326561 mon.0 10.252.24.25:6789/0 132 : [INF] pgmap v29066: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -12275> 2013-04-03 10:41:34.391834 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113648 ==== paxos(lease_ack lc 427417 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1615456900 0 0) 0x2787a00 con 0x208a840 -12274> 2013-04-03 10:41:34.391875 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos -12273> 2013-04-03 10:41:34.391893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos version 394065 summary v 394065 -12272> 2013-04-03 10:41:34.391910 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12271> 2013-04-03 10:41:35.486077 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113649 ==== forward(pg_stats(0 pgs tid 61 v 605) v1) to leader v1 ==== 368+0+0 (2941637205 0 0) 0x1bae8c0 con 0x208a840 -12270> 2013-04-03 10:41:35.486105 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 -12269> 2013-04-03 10:41:35.486117 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -12268> 2013-04-03 10:41:35.486171 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 preprocess_query pg_stats(0 pgs tid 61 v 605) v1 from osd.4 10.252.24.32:6800/13473 -12267> 2013-04-03 10:41:35.486196 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 prepare_update pg_stats(0 pgs tid 61 v 605) v1 from osd.4 10.252.24.32:6800/13473 -12266> 2013-04-03 10:41:35.486203 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 prepare_pg_stats pg_stats(0 pgs tid 61 v 605) v1 from osd.4 -12265> 2013-04-03 10:41:35.486209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 got osd.4 osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[])) -12264> 2013-04-03 10:41:35.536389 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 encode_pending v 29067 -12263> 2013-04-03 10:41:35.536427 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427417) queue_proposal bl 265 bytes; ctx = 0x2c3d620 -12262> 2013-04-03 10:41:35.536437 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427417) propose_queued 427418 265 bytes -12261> 2013-04-03 10:41:35.537705 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427417 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909000 -12260> 2013-04-03 10:41:35.566738 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113650 ==== paxos(accept lc 427417 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4200065436 0 0) 0x2909000 con 0x208a840 -12259> 2013-04-03 10:41:35.567040 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427418 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787a00 -12258> 2013-04-03 10:41:35.567070 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427418 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2787780 -12257> 2013-04-03 10:41:35.567142 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29066 update_from_paxos applying incremental 29067 -12256> 2013-04-03 10:41:35.567153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 v29067: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1844B/s wr, 0op/s -12255> 2013-04-03 10:41:35.567170 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 send_pg_creates to 0 pgs -12254> 2013-04-03 10:41:35.567176 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 update_logger -12253> 2013-04-03 10:41:35.567188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 create_pending v 29068 -12252> 2013-04-03 10:41:35.567200 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 _updated_stats for osd.4 10.252.24.32:6800/13473 -12251> 2013-04-03 10:41:35.567207 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(0 pgs tid 61) v1 tid 17623) v2 -- ?+0 0x24f8240 con 0x208a840 -12250> 2013-04-03 10:41:35.567217 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 check_osd_map already seen 605 >= 605 -12249> 2013-04-03 10:41:35.567220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 update_logger -12248> 2013-04-03 10:41:35.567231 7fcdffcdb700 0 log [INF] : pgmap v29067: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1844B/s wr, 0op/s -12247> 2013-04-03 10:41:35.567233 7fcdffcdb700 10 send_log to self -12246> 2013-04-03 10:41:35.567234 7fcdffcdb700 10 log_queue is 1 last_log 133 sent 132 num 1 unsent 1 sending 1 -12245> 2013-04-03 10:41:35.567235 7fcdffcdb700 10 will send 2013-04-03 10:41:35.567232 mon.0 10.252.24.25:6789/0 133 : [INF] pgmap v29067: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1844B/s wr, 0op/s -12244> 2013-04-03 10:41:35.567242 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75e00 -12243> 2013-04-03 10:41:35.567290 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos -12242> 2013-04-03 10:41:35.567309 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos version 394065 summary v 394065 -12241> 2013-04-03 10:41:35.567328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12240> 2013-04-03 10:41:35.567350 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75e00 con 0x1bac160 -12239> 2013-04-03 10:41:35.567371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos -12238> 2013-04-03 10:41:35.567386 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos version 394065 summary v 394065 -12237> 2013-04-03 10:41:35.567396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12236> 2013-04-03 10:41:35.567408 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 preprocess_log log(1 entries) v1 from mon.0 -12235> 2013-04-03 10:41:35.567433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12234> 2013-04-03 10:41:35.567449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 prepare_log log(1 entries) v1 from mon.0 -12233> 2013-04-03 10:41:35.567460 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 logging 2013-04-03 10:41:35.567232 mon.0 10.252.24.25:6789/0 133 : [INF] pgmap v29067: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1844B/s wr, 0op/s -12232> 2013-04-03 10:41:35.580398 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113651 ==== paxos(lease_ack lc 427418 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (315927494 0 0) 0x2787780 con 0x208a840 -12231> 2013-04-03 10:41:35.580438 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos -12230> 2013-04-03 10:41:35.580455 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 update_from_paxos version 394065 summary v 394065 -12229> 2013-04-03 10:41:35.580472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12228> 2013-04-03 10:41:35.589598 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 122 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140407 v187) v2 ==== 289+0+0 (252521111 0 0) 0x2ce12c0 con 0x2118420 -12227> 2013-04-03 10:41:35.589652 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140407 v187) v2 from mds.? 10.252.58.31:6800/10908 -12226> 2013-04-03 10:41:35.589664 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140407 v187) v2 -- ?+0 0x1bae8c0 con 0x2118420 -12225> 2013-04-03 10:41:36.566950 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394065 encode_pending v394066 -12224> 2013-04-03 10:41:36.566981 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427418) queue_proposal bl 443 bytes; ctx = 0x2c3d6a0 -12223> 2013-04-03 10:41:36.566989 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427418) propose_queued 427419 443 bytes -12222> 2013-04-03 10:41:36.568089 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427418 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909a00 -12221> 2013-04-03 10:41:36.595869 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113652 ==== paxos(accept lc 427418 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2551043638 0 0) 0x2909a00 con 0x208a840 -12220> 2013-04-03 10:41:36.596167 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427419 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2787780 -12219> 2013-04-03 10:41:36.596199 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427419 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2909000 -12218> 2013-04-03 10:41:36.596235 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos -12217> 2013-04-03 10:41:36.596263 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos version 394066 summary v 394065 -12216> 2013-04-03 10:41:36.596278 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos latest full 393798 -12215> 2013-04-03 10:41:36.596296 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos applying incremental log 394066 2013-04-03 10:41:35.567232 mon.0 10.252.24.25:6789/0 133 : [INF] pgmap v29067: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1844B/s wr, 0op/s -12214> 2013-04-03 10:41:36.596349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 check_subs -12213> 2013-04-03 10:41:36.596380 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 create_pending v 394067 -12212> 2013-04-03 10:41:36.596404 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 _updated_log for mon.0 10.252.24.25:6789/0 -12211> 2013-04-03 10:41:36.596420 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 133) v1 -- ?+0 0x2224c40 con 0x1bac160 -12210> 2013-04-03 10:41:36.596471 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos -12209> 2013-04-03 10:41:36.596488 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos version 394066 summary v 394066 -12208> 2013-04-03 10:41:36.596506 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12207> 2013-04-03 10:41:36.596529 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 133) v1 ==== 0+0+0 (0 0 0) 0x2224c40 con 0x1bac160 -12206> 2013-04-03 10:41:36.596537 7fcdffcdb700 10 handle_log_ack log(last 133) v1 -12205> 2013-04-03 10:41:36.596538 7fcdffcdb700 10 logged 2013-04-03 10:41:35.567232 mon.0 10.252.24.25:6789/0 133 : [INF] pgmap v29067: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1844B/s wr, 0op/s -12204> 2013-04-03 10:41:36.621910 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113653 ==== paxos(lease_ack lc 427419 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2507568711 0 0) 0x2909000 con 0x208a840 -12203> 2013-04-03 10:41:36.621947 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos -12202> 2013-04-03 10:41:36.621964 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos version 394066 summary v 394066 -12201> 2013-04-03 10:41:36.621981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12200> 2013-04-03 10:41:36.830808 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113654 ==== forward(pg_stats(0 pgs tid 410 v 605) v1) to leader v1 ==== 384+0+0 (30818757 0 0) 0x1baeb80 con 0x208a840 -12199> 2013-04-03 10:41:36.830822 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -12198> 2013-04-03 10:41:36.830832 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -12197> 2013-04-03 10:41:36.830863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 preprocess_query pg_stats(0 pgs tid 410 v 605) v1 from osd.0 10.252.58.31:6802/2682 -12196> 2013-04-03 10:41:36.830885 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 prepare_update pg_stats(0 pgs tid 410 v 605) v1 from osd.0 10.252.58.31:6802/2682 -12195> 2013-04-03 10:41:36.830891 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 prepare_pg_stats pg_stats(0 pgs tid 410 v 605) v1 from osd.0 -12194> 2013-04-03 10:41:36.830897 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 got osd.0 osd_stat(1140 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1139 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -12193> 2013-04-03 10:41:36.935835 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113655 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 122 v187) v2) to leader v1 ==== 560+0+0 (2868477768 0 0) 0x2a51000 con 0x208a840 -12192> 2013-04-03 10:41:36.935850 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -12191> 2013-04-03 10:41:36.935858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a51b00 from 10.252.24.88:6789/0 -12190> 2013-04-03 10:41:36.935889 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 122 v187) v2 from mds.0 10.252.24.25:6802/3144 -12189> 2013-04-03 10:41:36.935900 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 122 v187) v2 tid 17625) v2 -- ?+0 0x24f8b40 con 0x208a840 -12188> 2013-04-03 10:41:37.388406 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 122 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21778 v187) v2 ==== 289+0+0 (3702205389 0 0) 0x2a83840 con 0x200d420 -12187> 2013-04-03 10:41:37.388484 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21778 v187) v2 from mds.? 10.252.24.88:6800/17213 -12186> 2013-04-03 10:41:37.388498 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21778 v187) v2 -- ?+0 0x2a51000 con 0x200d420 -12185> 2013-04-03 10:41:37.596066 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 encode_pending v 29068 -12184> 2013-04-03 10:41:37.596099 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427419) queue_proposal bl 281 bytes; ctx = 0x2c3d520 -12183> 2013-04-03 10:41:37.596107 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427419) propose_queued 427420 281 bytes -12182> 2013-04-03 10:41:37.597909 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427419 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909280 -12181> 2013-04-03 10:41:37.621994 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113656 ==== paxos(accept lc 427419 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3447787200 0 0) 0x2909280 con 0x208a840 -12180> 2013-04-03 10:41:37.634097 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427420 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909000 -12179> 2013-04-03 10:41:37.634136 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427420 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2909a00 -12178> 2013-04-03 10:41:37.634209 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29067 update_from_paxos applying incremental 29068 -12177> 2013-04-03 10:41:37.634221 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 v29068: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail -12176> 2013-04-03 10:41:37.634241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 send_pg_creates to 0 pgs -12175> 2013-04-03 10:41:37.634246 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 update_logger -12174> 2013-04-03 10:41:37.634257 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 create_pending v 29069 -12173> 2013-04-03 10:41:37.634266 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 _updated_stats for osd.0 10.252.58.31:6802/2682 -12172> 2013-04-03 10:41:37.634274 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(0 pgs tid 410) v1 tid 17624) v2 -- ?+0 0x24f8900 con 0x208a840 -12171> 2013-04-03 10:41:37.634284 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 check_osd_map already seen 605 >= 605 -12170> 2013-04-03 10:41:37.634287 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 update_logger -12169> 2013-04-03 10:41:37.634303 7fcdffcdb700 0 log [INF] : pgmap v29068: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail -12168> 2013-04-03 10:41:37.634306 7fcdffcdb700 10 send_log to self -12167> 2013-04-03 10:41:37.634306 7fcdffcdb700 10 log_queue is 1 last_log 134 sent 133 num 1 unsent 1 sending 1 -12166> 2013-04-03 10:41:37.634308 7fcdffcdb700 10 will send 2013-04-03 10:41:37.634305 mon.0 10.252.24.25:6789/0 134 : [INF] pgmap v29068: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail -12165> 2013-04-03 10:41:37.634314 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75e00 -12164> 2013-04-03 10:41:37.634365 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos -12163> 2013-04-03 10:41:37.634385 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos version 394066 summary v 394066 -12162> 2013-04-03 10:41:37.634404 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12161> 2013-04-03 10:41:37.634427 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75e00 con 0x1bac160 -12160> 2013-04-03 10:41:37.634449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos -12159> 2013-04-03 10:41:37.634464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos version 394066 summary v 394066 -12158> 2013-04-03 10:41:37.634474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12157> 2013-04-03 10:41:37.634486 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 preprocess_log log(1 entries) v1 from mon.0 -12156> 2013-04-03 10:41:37.634511 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12155> 2013-04-03 10:41:37.634525 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 prepare_log log(1 entries) v1 from mon.0 -12154> 2013-04-03 10:41:37.635252 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 logging 2013-04-03 10:41:37.634305 mon.0 10.252.24.25:6789/0 134 : [INF] pgmap v29068: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail -12153> 2013-04-03 10:41:37.644025 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113657 ==== paxos(lease_ack lc 427420 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3084780134 0 0) 0x2909a00 con 0x208a840 -12152> 2013-04-03 10:41:37.644062 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos -12151> 2013-04-03 10:41:37.644079 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos version 394066 summary v 394066 -12150> 2013-04-03 10:41:37.644096 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12149> 2013-04-03 10:41:38.357041 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 check_down_pgs -12148> 2013-04-03 10:41:38.357078 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 v29068: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail -12147> 2013-04-03 10:41:38.357106 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -12146> 2013-04-03 10:41:38.357131 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -12145> 2013-04-03 10:41:38.357169 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -12144> 2013-04-03 10:41:38.357201 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos -12143> 2013-04-03 10:41:38.357220 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 update_from_paxos version 394066 summary v 394066 -12142> 2013-04-03 10:41:38.357230 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 log -12141> 2013-04-03 10:41:38.357247 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12140> 2013-04-03 10:41:38.357266 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -12139> 2013-04-03 10:41:38.622192 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394066 encode_pending v394067 -12138> 2013-04-03 10:41:38.622222 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427420) queue_proposal bl 424 bytes; ctx = 0x2c3d650 -12137> 2013-04-03 10:41:38.622230 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427420) propose_queued 427421 424 bytes -12136> 2013-04-03 10:41:38.623356 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427420 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909500 -12135> 2013-04-03 10:41:38.650552 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113658 ==== paxos(accept lc 427420 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2696745801 0 0) 0x2909500 con 0x208a840 -12134> 2013-04-03 10:41:38.651499 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427421 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909a00 -12133> 2013-04-03 10:41:38.651529 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427421 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2909280 -12132> 2013-04-03 10:41:38.651568 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos -12131> 2013-04-03 10:41:38.651596 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos version 394067 summary v 394066 -12130> 2013-04-03 10:41:38.651612 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos latest full 393798 -12129> 2013-04-03 10:41:38.651631 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos applying incremental log 394067 2013-04-03 10:41:37.634305 mon.0 10.252.24.25:6789/0 134 : [INF] pgmap v29068: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail -12128> 2013-04-03 10:41:38.651679 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 check_subs -12127> 2013-04-03 10:41:38.651711 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 create_pending v 394068 -12126> 2013-04-03 10:41:38.651734 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 _updated_log for mon.0 10.252.24.25:6789/0 -12125> 2013-04-03 10:41:38.651747 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 134) v1 -- ?+0 0x2224540 con 0x1bac160 -12124> 2013-04-03 10:41:38.651799 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos -12123> 2013-04-03 10:41:38.651815 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos version 394067 summary v 394067 -12122> 2013-04-03 10:41:38.651834 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12121> 2013-04-03 10:41:38.651857 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 134) v1 ==== 0+0+0 (0 0 0) 0x2224540 con 0x1bac160 -12120> 2013-04-03 10:41:38.651865 7fcdffcdb700 10 handle_log_ack log(last 134) v1 -12119> 2013-04-03 10:41:38.651867 7fcdffcdb700 10 logged 2013-04-03 10:41:37.634305 mon.0 10.252.24.25:6789/0 134 : [INF] pgmap v29068: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail -12118> 2013-04-03 10:41:38.662876 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113659 ==== paxos(lease_ack lc 427421 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (106821834 0 0) 0x2909280 con 0x208a840 -12117> 2013-04-03 10:41:38.662915 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos -12116> 2013-04-03 10:41:38.662933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos version 394067 summary v 394067 -12115> 2013-04-03 10:41:38.662948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12114> 2013-04-03 10:41:39.589734 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 123 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140408 v187) v2 ==== 289+0+0 (4187855869 0 0) 0x2ce1580 con 0x2118420 -12113> 2013-04-03 10:41:39.589821 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140408 v187) v2 from mds.? 10.252.58.31:6800/10908 -12112> 2013-04-03 10:41:39.589835 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140408 v187) v2 -- ?+0 0x2a83840 con 0x2118420 -12111> 2013-04-03 10:41:40.935995 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113660 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 123 v187) v2) to leader v1 ==== 560+0+0 (999830319 0 0) 0x1baeb80 con 0x208a840 -12110> 2013-04-03 10:41:40.936017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -12109> 2013-04-03 10:41:40.936028 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a52600 from 10.252.24.88:6789/0 -12108> 2013-04-03 10:41:40.936081 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 123 v187) v2 from mds.0 10.252.24.25:6802/3144 -12107> 2013-04-03 10:41:40.936094 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 123 v187) v2 tid 17626) v2 -- ?+0 0x24f8240 con 0x208a840 -12106> 2013-04-03 10:41:41.388512 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 123 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21779 v187) v2 ==== 289+0+0 (4291138736 0 0) 0x2a848c0 con 0x200d420 -12105> 2013-04-03 10:41:41.388583 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21779 v187) v2 from mds.? 10.252.24.88:6800/17213 -12104> 2013-04-03 10:41:41.388597 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21779 v187) v2 -- ?+0 0x1baeb80 con 0x200d420 -12103> 2013-04-03 10:41:41.603714 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113661 ==== forward(pg_stats(1 pgs tid 40 v 605) v1) to leader v1 ==== 753+0+0 (346259987 0 0) 0x2ce1580 con 0x208a840 -12102> 2013-04-03 10:41:41.603737 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -12101> 2013-04-03 10:41:41.603747 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -12100> 2013-04-03 10:41:41.603797 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 preprocess_query pg_stats(1 pgs tid 40 v 605) v1 from osd.1 10.252.24.25:6805/3572 -12099> 2013-04-03 10:41:41.603821 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 prepare_update pg_stats(1 pgs tid 40 v 605) v1 from osd.1 10.252.24.25:6805/3572 -12098> 2013-04-03 10:41:41.603829 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 prepare_pg_stats pg_stats(1 pgs tid 40 v 605) v1 from osd.1 -12097> 2013-04-03 10:41:41.603836 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -12096> 2013-04-03 10:41:41.651598 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427421 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2909c80 -12095> 2013-04-03 10:41:41.652328 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113662 ==== paxos(lease_ack lc 427421 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3205866015 0 0) 0x2909c80 con 0x208a840 -12094> 2013-04-03 10:41:41.652374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos -12093> 2013-04-03 10:41:41.652394 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos version 394067 summary v 394067 -12092> 2013-04-03 10:41:41.652411 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12091> 2013-04-03 10:41:41.654014 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 encode_pending v 29069 -12090> 2013-04-03 10:41:41.654048 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427421) queue_proposal bl 650 bytes; ctx = 0x2c3d4f0 -12089> 2013-04-03 10:41:41.654057 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427421) propose_queued 427422 650 bytes -12088> 2013-04-03 10:41:41.655544 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427421 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1a00 -12087> 2013-04-03 10:41:41.687122 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113663 ==== paxos(accept lc 427421 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (903478078 0 0) 0x29a1a00 con 0x208a840 -12086> 2013-04-03 10:41:41.692316 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427422 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909c80 -12085> 2013-04-03 10:41:41.692345 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427422 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2909280 -12084> 2013-04-03 10:41:41.692415 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29068 update_from_paxos applying incremental 29069 -12083> 2013-04-03 10:41:41.692430 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 v29069: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 669B/s wr, 0op/s -12082> 2013-04-03 10:41:41.692449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 send_pg_creates to 0 pgs -12081> 2013-04-03 10:41:41.692454 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 update_logger -12080> 2013-04-03 10:41:41.692464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 create_pending v 29070 -12079> 2013-04-03 10:41:41.692475 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 _updated_stats for osd.1 10.252.24.25:6805/3572 -12078> 2013-04-03 10:41:41.692482 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 40) v1 tid 17627) v2 -- ?+0 0x24f8900 con 0x208a840 -12077> 2013-04-03 10:41:41.692491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 check_osd_map already seen 605 >= 605 -12076> 2013-04-03 10:41:41.692493 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 update_logger -12075> 2013-04-03 10:41:41.692504 7fcdffcdb700 0 log [INF] : pgmap v29069: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 669B/s wr, 0op/s -12074> 2013-04-03 10:41:41.692512 7fcdffcdb700 10 send_log to self -12073> 2013-04-03 10:41:41.692513 7fcdffcdb700 10 log_queue is 1 last_log 135 sent 134 num 1 unsent 1 sending 1 -12072> 2013-04-03 10:41:41.692515 7fcdffcdb700 10 will send 2013-04-03 10:41:41.692512 mon.0 10.252.24.25:6789/0 135 : [INF] pgmap v29069: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 669B/s wr, 0op/s -12071> 2013-04-03 10:41:41.692521 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75e00 -12070> 2013-04-03 10:41:41.692569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos -12069> 2013-04-03 10:41:41.692587 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos version 394067 summary v 394067 -12068> 2013-04-03 10:41:41.692604 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12067> 2013-04-03 10:41:41.692624 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75e00 con 0x1bac160 -12066> 2013-04-03 10:41:41.692645 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos -12065> 2013-04-03 10:41:41.692660 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos version 394067 summary v 394067 -12064> 2013-04-03 10:41:41.692669 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12063> 2013-04-03 10:41:41.692679 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 preprocess_log log(1 entries) v1 from mon.0 -12062> 2013-04-03 10:41:41.692702 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12061> 2013-04-03 10:41:41.692714 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 prepare_log log(1 entries) v1 from mon.0 -12060> 2013-04-03 10:41:41.692724 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 logging 2013-04-03 10:41:41.692512 mon.0 10.252.24.25:6789/0 135 : [INF] pgmap v29069: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 669B/s wr, 0op/s -12059> 2013-04-03 10:41:41.721571 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113664 ==== paxos(lease_ack lc 427422 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3741711385 0 0) 0x2909280 con 0x208a840 -12058> 2013-04-03 10:41:41.721610 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos -12057> 2013-04-03 10:41:41.721627 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 update_from_paxos version 394067 summary v 394067 -12056> 2013-04-03 10:41:41.721642 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12055> 2013-04-03 10:41:41.769162 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113665 ==== forward(pg_stats(1 pgs tid 30452 v 605) v1) to leader v1 ==== 757+0+0 (325416022 0 0) 0x2a512c0 con 0x208a840 -12054> 2013-04-03 10:41:41.769174 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -12053> 2013-04-03 10:41:41.769181 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -12052> 2013-04-03 10:41:41.769209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 preprocess_query pg_stats(1 pgs tid 30452 v 605) v1 from osd.3 10.252.24.88:6801/17538 -12051> 2013-04-03 10:41:41.769229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 prepare_update pg_stats(1 pgs tid 30452 v 605) v1 from osd.3 10.252.24.88:6801/17538 -12050> 2013-04-03 10:41:41.769235 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 prepare_pg_stats pg_stats(1 pgs tid 30452 v 605) v1 from osd.3 -12049> 2013-04-03 10:41:41.769241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -12048> 2013-04-03 10:41:42.687316 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 encode_pending v 29070 -12047> 2013-04-03 10:41:42.687350 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427422) queue_proposal bl 654 bytes; ctx = 0x2c3d350 -12046> 2013-04-03 10:41:42.687358 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427422) propose_queued 427423 654 bytes -12045> 2013-04-03 10:41:42.688410 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427422 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1000 -12044> 2013-04-03 10:41:42.688510 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394067 encode_pending v394068 -12043> 2013-04-03 10:41:42.688541 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427382..427422) queue_proposal bl 442 bytes; ctx = 0x2c3d2d0 -12042> 2013-04-03 10:41:42.688548 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427382..427422) propose_new_value not active; proposal queued -12041> 2013-04-03 10:41:42.713311 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113666 ==== paxos(accept lc 427422 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1612915457 0 0) 0x29a1000 con 0x208a840 -12040> 2013-04-03 10:41:42.713632 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427423 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909280 -12039> 2013-04-03 10:41:42.713659 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427423 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29a1a00 -12038> 2013-04-03 10:41:42.713733 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29069 update_from_paxos applying incremental 29070 -12037> 2013-04-03 10:41:42.713752 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 v29070: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1005B/s wr, 0op/s -12036> 2013-04-03 10:41:42.713772 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 send_pg_creates to 0 pgs -12035> 2013-04-03 10:41:42.713777 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 update_logger -12034> 2013-04-03 10:41:42.713787 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 create_pending v 29071 -12033> 2013-04-03 10:41:42.713797 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 _updated_stats for osd.3 10.252.24.88:6801/17538 -12032> 2013-04-03 10:41:42.713804 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30452) v1 tid 17628) v2 -- ?+0 0x24f8240 con 0x208a840 -12031> 2013-04-03 10:41:42.713821 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 check_osd_map already seen 605 >= 605 -12030> 2013-04-03 10:41:42.713825 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 update_logger -12029> 2013-04-03 10:41:42.713842 7fcdffcdb700 0 log [INF] : pgmap v29070: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1005B/s wr, 0op/s -12028> 2013-04-03 10:41:42.713845 7fcdffcdb700 10 send_log to self -12027> 2013-04-03 10:41:42.713846 7fcdffcdb700 10 log_queue is 2 last_log 136 sent 135 num 2 unsent 1 sending 1 -12026> 2013-04-03 10:41:42.713848 7fcdffcdb700 10 will send 2013-04-03 10:41:41.692512 mon.0 10.252.24.25:6789/0 135 : [INF] pgmap v29069: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 669B/s wr, 0op/s -12025> 2013-04-03 10:41:42.713855 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74800 -12024> 2013-04-03 10:41:42.713881 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427423) propose_queued 427424 442 bytes -12023> 2013-04-03 10:41:42.714098 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427423 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909500 -12022> 2013-04-03 10:41:42.714130 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74800 con 0x1bac160 -12021> 2013-04-03 10:41:42.726967 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113667 ==== paxos(lease_ack lc 427423 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1676060313 0 0) 0x2909500 con 0x208a840 -12020> 2013-04-03 10:41:42.739218 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113668 ==== paxos(accept lc 427423 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (642124544 0 0) 0x29a1a00 con 0x208a840 -12019> 2013-04-03 10:41:42.740358 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427424 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2909500 -12018> 2013-04-03 10:41:42.740387 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427424 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29a1000 -12017> 2013-04-03 10:41:42.740423 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos -12016> 2013-04-03 10:41:42.740449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394067 -12015> 2013-04-03 10:41:42.740464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos latest full 393798 -12014> 2013-04-03 10:41:42.740482 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos applying incremental log 394068 2013-04-03 10:41:41.692512 mon.0 10.252.24.25:6789/0 135 : [INF] pgmap v29069: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 669B/s wr, 0op/s -12013> 2013-04-03 10:41:42.740526 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 check_subs -12012> 2013-04-03 10:41:42.740556 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 create_pending v 394069 -12011> 2013-04-03 10:41:42.741118 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 _updated_log for mon.0 10.252.24.25:6789/0 -12010> 2013-04-03 10:41:42.741137 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 135) v1 -- ?+0 0x22248c0 con 0x1bac160 -12009> 2013-04-03 10:41:42.741157 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos -12008> 2013-04-03 10:41:42.741173 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 -12007> 2013-04-03 10:41:42.741183 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -12006> 2013-04-03 10:41:42.741195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 preprocess_log log(1 entries) v1 from mon.0 -12005> 2013-04-03 10:41:42.741216 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 nothing new -12004> 2013-04-03 10:41:42.741267 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos -12003> 2013-04-03 10:41:42.741283 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 -12002> 2013-04-03 10:41:42.741301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -12001> 2013-04-03 10:41:42.741331 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 135) v1 ==== 0+0+0 (0 0 0) 0x22248c0 con 0x1bac160 -12000> 2013-04-03 10:41:42.741340 7fcdffcdb700 10 handle_log_ack log(last 135) v1 -11999> 2013-04-03 10:41:42.741342 7fcdffcdb700 10 logged 2013-04-03 10:41:41.692512 mon.0 10.252.24.25:6789/0 135 : [INF] pgmap v29069: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 669B/s wr, 0op/s -11998> 2013-04-03 10:41:42.790455 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113669 ==== paxos(lease_ack lc 427424 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1401051281 0 0) 0x29a1000 con 0x208a840 -11997> 2013-04-03 10:41:42.790501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos -11996> 2013-04-03 10:41:42.790519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 -11995> 2013-04-03 10:41:42.790536 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11994> 2013-04-03 10:41:43.357417 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 check_down_pgs -11993> 2013-04-03 10:41:43.357448 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 v29070: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1005B/s wr, 0op/s -11992> 2013-04-03 10:41:43.357477 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -11991> 2013-04-03 10:41:43.357502 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -11990> 2013-04-03 10:41:43.357540 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -11989> 2013-04-03 10:41:43.357571 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos -11988> 2013-04-03 10:41:43.357590 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 -11987> 2013-04-03 10:41:43.357600 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 log -11986> 2013-04-03 10:41:43.357622 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11985> 2013-04-03 10:41:43.357641 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -11984> 2013-04-03 10:41:43.589869 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 124 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140409 v187) v2 ==== 289+0+0 (3673367680 0 0) 0x2ce1840 con 0x2118420 -11983> 2013-04-03 10:41:43.589946 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140409 v187) v2 from mds.? 10.252.58.31:6800/10908 -11982> 2013-04-03 10:41:43.589961 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140409 v187) v2 -- ?+0 0x2a512c0 con 0x2118420 -11981> 2013-04-03 10:41:44.936203 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113670 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 124 v187) v2) to leader v1 ==== 560+0+0 (90515750 0 0) 0x2a52080 con 0x208a840 -11980> 2013-04-03 10:41:44.936223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11979> 2013-04-03 10:41:44.936234 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2996080 from 10.252.24.88:6789/0 -11978> 2013-04-03 10:41:44.936287 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 124 v187) v2 from mds.0 10.252.24.25:6802/3144 -11977> 2013-04-03 10:41:44.936299 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 124 v187) v2 tid 17629) v2 -- ?+0 0x24f8900 con 0x208a840 -11976> 2013-04-03 10:41:45.388619 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 124 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21780 v187) v2 ==== 289+0+0 (399288515 0 0) 0x2a832c0 con 0x200d420 -11975> 2013-04-03 10:41:45.388694 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21780 v187) v2 from mds.? 10.252.24.88:6800/17213 -11974> 2013-04-03 10:41:45.388712 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21780 v187) v2 -- ?+0 0x2a52080 con 0x200d420 -11973> 2013-04-03 10:41:45.740462 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427424 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29a1500 -11972> 2013-04-03 10:41:45.741265 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113671 ==== paxos(lease_ack lc 427424 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1725687349 0 0) 0x29a1500 con 0x208a840 -11971> 2013-04-03 10:41:45.741338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos -11970> 2013-04-03 10:41:45.741359 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 -11969> 2013-04-03 10:41:45.741377 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11968> 2013-04-03 10:41:47.589975 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 125 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140410 v187) v2 ==== 289+0+0 (3208701191 0 0) 0x2efab80 con 0x2118420 -11967> 2013-04-03 10:41:47.590055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140410 v187) v2 from mds.? 10.252.58.31:6800/10908 -11966> 2013-04-03 10:41:47.590069 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140410 v187) v2 -- ?+0 0x2a832c0 con 0x2118420 -11965> 2013-04-03 10:41:48.247260 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113672 ==== forward(pg_stats(1 pgs tid 20 v 605) v1) to leader v1 ==== 757+0+0 (4129646956 0 0) 0x2ce1840 con 0x208a840 -11964> 2013-04-03 10:41:48.247281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -11963> 2013-04-03 10:41:48.247292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -11962> 2013-04-03 10:41:48.247342 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 preprocess_query pg_stats(1 pgs tid 20 v 605) v1 from osd.2 10.252.24.30:6801/5656 -11961> 2013-04-03 10:41:48.247369 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 prepare_update pg_stats(1 pgs tid 20 v 605) v1 from osd.2 10.252.24.30:6801/5656 -11960> 2013-04-03 10:41:48.247373 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 prepare_pg_stats pg_stats(1 pgs tid 20 v 605) v1 from osd.2 -11959> 2013-04-03 10:41:48.247380 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -11958> 2013-04-03 10:41:48.297520 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 encode_pending v 29071 -11957> 2013-04-03 10:41:48.297563 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427424) queue_proposal bl 654 bytes; ctx = 0x2c3d0f0 -11956> 2013-04-03 10:41:48.297573 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427424) propose_queued 427425 654 bytes -11955> 2013-04-03 10:41:48.298432 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427424 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1280 -11954> 2013-04-03 10:41:48.325855 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113673 ==== paxos(accept lc 427424 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (831999561 0 0) 0x29a1280 con 0x208a840 -11953> 2013-04-03 10:41:48.326213 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427425 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1500 -11952> 2013-04-03 10:41:48.326246 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427425 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29a1000 -11951> 2013-04-03 10:41:48.326326 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29070 update_from_paxos applying incremental 29071 -11950> 2013-04-03 10:41:48.326348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 v29071: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -11949> 2013-04-03 10:41:48.326367 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 send_pg_creates to 0 pgs -11948> 2013-04-03 10:41:48.326371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 update_logger -11947> 2013-04-03 10:41:48.326382 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 create_pending v 29072 -11946> 2013-04-03 10:41:48.326392 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 _updated_stats for osd.2 10.252.24.30:6801/5656 -11945> 2013-04-03 10:41:48.326400 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 20) v1 tid 17630) v2 -- ?+0 0x24f8240 con 0x208a840 -11944> 2013-04-03 10:41:48.326410 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 check_osd_map already seen 605 >= 605 -11943> 2013-04-03 10:41:48.326413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 update_logger -11942> 2013-04-03 10:41:48.326424 7fcdffcdb700 0 log [INF] : pgmap v29071: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -11941> 2013-04-03 10:41:48.326426 7fcdffcdb700 10 send_log to self -11940> 2013-04-03 10:41:48.326427 7fcdffcdb700 10 log_queue is 2 last_log 137 sent 135 num 2 unsent 2 sending 2 -11939> 2013-04-03 10:41:48.326429 7fcdffcdb700 10 will send 2013-04-03 10:41:42.713845 mon.0 10.252.24.25:6789/0 136 : [INF] pgmap v29070: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1005B/s wr, 0op/s -11938> 2013-04-03 10:41:48.326441 7fcdffcdb700 10 will send 2013-04-03 10:41:48.326425 mon.0 10.252.24.25:6789/0 137 : [INF] pgmap v29071: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -11937> 2013-04-03 10:41:48.326445 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b74800 -11936> 2013-04-03 10:41:48.326497 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos -11935> 2013-04-03 10:41:48.326518 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 -11934> 2013-04-03 10:41:48.326537 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11933> 2013-04-03 10:41:48.326558 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74800 con 0x1bac160 -11932> 2013-04-03 10:41:48.326580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos -11931> 2013-04-03 10:41:48.326596 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 -11930> 2013-04-03 10:41:48.326605 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -11929> 2013-04-03 10:41:48.326617 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 preprocess_log log(2 entries) v1 from mon.0 -11928> 2013-04-03 10:41:48.326649 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -11927> 2013-04-03 10:41:48.326662 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 prepare_log log(2 entries) v1 from mon.0 -11926> 2013-04-03 10:41:48.326674 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 logging 2013-04-03 10:41:42.713845 mon.0 10.252.24.25:6789/0 136 : [INF] pgmap v29070: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1005B/s wr, 0op/s -11925> 2013-04-03 10:41:48.326696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 logging 2013-04-03 10:41:48.326425 mon.0 10.252.24.25:6789/0 137 : [INF] pgmap v29071: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -11924> 2013-04-03 10:41:48.338187 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113674 ==== paxos(lease_ack lc 427425 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (4065614157 0 0) 0x29a1000 con 0x208a840 -11923> 2013-04-03 10:41:48.338229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos -11922> 2013-04-03 10:41:48.338246 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 -11921> 2013-04-03 10:41:48.338263 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11920> 2013-04-03 10:41:48.357758 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 check_down_pgs -11919> 2013-04-03 10:41:48.357797 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 v29071: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -11918> 2013-04-03 10:41:48.357821 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -11917> 2013-04-03 10:41:48.357844 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -11916> 2013-04-03 10:41:48.357882 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -11915> 2013-04-03 10:41:48.357913 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos -11914> 2013-04-03 10:41:48.357930 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 update_from_paxos version 394068 summary v 394068 -11913> 2013-04-03 10:41:48.357940 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 log -11912> 2013-04-03 10:41:48.357956 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11911> 2013-04-03 10:41:48.357973 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -11910> 2013-04-03 10:41:48.936248 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113675 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 125 v187) v2) to leader v1 ==== 560+0+0 (3624938611 0 0) 0x2995580 con 0x208a840 -11909> 2013-04-03 10:41:48.936271 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11908> 2013-04-03 10:41:48.936282 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2996b80 from 10.252.24.88:6789/0 -11907> 2013-04-03 10:41:48.936334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 125 v187) v2 from mds.0 10.252.24.25:6802/3144 -11906> 2013-04-03 10:41:48.936346 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 125 v187) v2 tid 17631) v2 -- ?+0 0x24f8900 con 0x208a840 -11905> 2013-04-03 10:41:49.326076 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394068 encode_pending v394069 -11904> 2013-04-03 10:41:49.326107 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427425) queue_proposal bl 726 bytes; ctx = 0x2c3d600 -11903> 2013-04-03 10:41:49.326114 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427425) propose_queued 427426 726 bytes -11902> 2013-04-03 10:41:49.327728 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427425 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1c80 -11901> 2013-04-03 10:41:49.357281 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113676 ==== paxos(accept lc 427425 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2763791946 0 0) 0x29a1c80 con 0x208a840 -11900> 2013-04-03 10:41:49.357637 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427426 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29a1000 -11899> 2013-04-03 10:41:49.357667 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427426 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29a1280 -11898> 2013-04-03 10:41:49.357706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos -11897> 2013-04-03 10:41:49.357733 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394068 -11896> 2013-04-03 10:41:49.357751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos latest full 393798 -11895> 2013-04-03 10:41:49.357770 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos applying incremental log 394069 2013-04-03 10:41:42.713845 mon.0 10.252.24.25:6789/0 136 : [INF] pgmap v29070: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1005B/s wr, 0op/s -11894> 2013-04-03 10:41:49.357799 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos applying incremental log 394069 2013-04-03 10:41:48.326425 mon.0 10.252.24.25:6789/0 137 : [INF] pgmap v29071: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -11893> 2013-04-03 10:41:49.357844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 check_subs -11892> 2013-04-03 10:41:49.358218 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 create_pending v 394070 -11891> 2013-04-03 10:41:49.358245 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 _updated_log for mon.0 10.252.24.25:6789/0 -11890> 2013-04-03 10:41:49.358259 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 137) v1 -- ?+0 0x2224e00 con 0x1bac160 -11889> 2013-04-03 10:41:49.358313 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos -11888> 2013-04-03 10:41:49.358330 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 -11887> 2013-04-03 10:41:49.358349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11886> 2013-04-03 10:41:49.358374 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 137) v1 ==== 0+0+0 (0 0 0) 0x2224e00 con 0x1bac160 -11885> 2013-04-03 10:41:49.358382 7fcdffcdb700 10 handle_log_ack log(last 137) v1 -11884> 2013-04-03 10:41:49.358383 7fcdffcdb700 10 logged 2013-04-03 10:41:42.713845 mon.0 10.252.24.25:6789/0 136 : [INF] pgmap v29070: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1005B/s wr, 0op/s -11883> 2013-04-03 10:41:49.358390 7fcdffcdb700 10 logged 2013-04-03 10:41:48.326425 mon.0 10.252.24.25:6789/0 137 : [INF] pgmap v29071: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -11882> 2013-04-03 10:41:49.382483 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113677 ==== paxos(lease_ack lc 427426 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2721439686 0 0) 0x29a1280 con 0x208a840 -11881> 2013-04-03 10:41:49.382523 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos -11880> 2013-04-03 10:41:49.382540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 -11879> 2013-04-03 10:41:49.382556 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11878> 2013-04-03 10:41:49.388681 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 125 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21781 v187) v2 ==== 289+0+0 (883038142 0 0) 0x2145000 con 0x200d420 -11877> 2013-04-03 10:41:49.388731 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21781 v187) v2 from mds.? 10.252.24.88:6800/17213 -11876> 2013-04-03 10:41:49.388743 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21781 v187) v2 -- ?+0 0x2995580 con 0x200d420 -11875> 2013-04-03 10:41:50.809192 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2df2280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33093/0 -11874> 2013-04-03 10:41:50.809217 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33093/0) -11873> 2013-04-03 10:41:50.809235 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2280 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -11872> 2013-04-03 10:41:50.809449 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2280 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -11871> 2013-04-03 10:41:51.590126 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 126 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140411 v187) v2 ==== 289+0+0 (2620283514 0 0) 0x2efa8c0 con 0x2118420 -11870> 2013-04-03 10:41:51.590204 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140411 v187) v2 from mds.? 10.252.58.31:6800/10908 -11869> 2013-04-03 10:41:51.590218 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140411 v187) v2 -- ?+0 0x2145000 con 0x2118420 -11868> 2013-04-03 10:41:52.357741 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427426 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2c4dc80 -11867> 2013-04-03 10:41:52.358504 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113678 ==== paxos(lease_ack lc 427426 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3847480146 0 0) 0x2c4dc80 con 0x208a840 -11866> 2013-04-03 10:41:52.358580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos -11865> 2013-04-03 10:41:52.358601 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 -11864> 2013-04-03 10:41:52.358619 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11863> 2013-04-03 10:41:52.936351 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113679 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 126 v187) v2) to leader v1 ==== 560+0+0 (3825080288 0 0) 0x2ce1840 con 0x208a840 -11862> 2013-04-03 10:41:52.936374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11861> 2013-04-03 10:41:52.936387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29952c0 from 10.252.24.88:6789/0 -11860> 2013-04-03 10:41:52.936433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 126 v187) v2 from mds.0 10.252.24.25:6802/3144 -11859> 2013-04-03 10:41:52.936445 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 126 v187) v2 tid 17632) v2 -- ?+0 0x24f8d80 con 0x208a840 -11858> 2013-04-03 10:41:53.358124 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 v29071: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -11857> 2013-04-03 10:41:53.358170 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -11856> 2013-04-03 10:41:53.358193 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -11855> 2013-04-03 10:41:53.358235 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -11854> 2013-04-03 10:41:53.358266 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos -11853> 2013-04-03 10:41:53.358286 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 -11852> 2013-04-03 10:41:53.358295 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 log -11851> 2013-04-03 10:41:53.358312 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11850> 2013-04-03 10:41:53.358332 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -11849> 2013-04-03 10:41:53.388822 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 126 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21782 v187) v2 ==== 289+0+0 (1360123449 0 0) 0x2cc5600 con 0x200d420 -11848> 2013-04-03 10:41:53.388880 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21782 v187) v2 from mds.? 10.252.24.88:6800/17213 -11847> 2013-04-03 10:41:53.388892 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21782 v187) v2 -- ?+0 0x2ce1840 con 0x200d420 -11846> 2013-04-03 10:41:55.357815 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427426 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2c4da00 -11845> 2013-04-03 10:41:55.358642 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113680 ==== paxos(lease_ack lc 427426 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2414924828 0 0) 0x2c4da00 con 0x208a840 -11844> 2013-04-03 10:41:55.358716 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos -11843> 2013-04-03 10:41:55.358738 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 -11842> 2013-04-03 10:41:55.358755 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11841> 2013-04-03 10:41:55.590264 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 127 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140412 v187) v2 ==== 289+0+0 (1948739081 0 0) 0x2efa600 con 0x2118420 -11840> 2013-04-03 10:41:55.590338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140412 v187) v2 from mds.? 10.252.58.31:6800/10908 -11839> 2013-04-03 10:41:55.590352 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140412 v187) v2 -- ?+0 0x2cc5600 con 0x2118420 -11838> 2013-04-03 10:41:56.605586 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113681 ==== forward(pg_stats(1 pgs tid 41 v 605) v1) to leader v1 ==== 753+0+0 (3007947174 0 0) 0x2efa8c0 con 0x208a840 -11837> 2013-04-03 10:41:56.605609 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -11836> 2013-04-03 10:41:56.605620 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -11835> 2013-04-03 10:41:56.605672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 preprocess_query pg_stats(1 pgs tid 41 v 605) v1 from osd.1 10.252.24.25:6805/3572 -11834> 2013-04-03 10:41:56.605697 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 prepare_update pg_stats(1 pgs tid 41 v 605) v1 from osd.1 10.252.24.25:6805/3572 -11833> 2013-04-03 10:41:56.605703 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 prepare_pg_stats pg_stats(1 pgs tid 41 v 605) v1 from osd.1 -11832> 2013-04-03 10:41:56.605710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -11831> 2013-04-03 10:41:56.656240 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 encode_pending v 29072 -11830> 2013-04-03 10:41:56.656279 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427426) queue_proposal bl 650 bytes; ctx = 0x2c3d740 -11829> 2013-04-03 10:41:56.656288 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427426) propose_queued 427427 650 bytes -11828> 2013-04-03 10:41:56.657919 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427426 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d780 -11827> 2013-04-03 10:41:56.683876 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113682 ==== paxos(accept lc 427426 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2856324901 0 0) 0x2c4d780 con 0x208a840 -11826> 2013-04-03 10:41:56.684567 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427427 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4da00 -11825> 2013-04-03 10:41:56.684586 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427427 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2c4dc80 -11824> 2013-04-03 10:41:56.684654 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29071 update_from_paxos applying incremental 29072 -11823> 2013-04-03 10:41:56.684671 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 v29072: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -11822> 2013-04-03 10:41:56.684691 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 send_pg_creates to 0 pgs -11821> 2013-04-03 10:41:56.684696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 update_logger -11820> 2013-04-03 10:41:56.684706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 create_pending v 29073 -11819> 2013-04-03 10:41:56.684716 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 _updated_stats for osd.1 10.252.24.25:6805/3572 -11818> 2013-04-03 10:41:56.684721 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 41) v1 tid 17633) v2 -- ?+0 0x24f8900 con 0x208a840 -11817> 2013-04-03 10:41:56.684731 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 check_osd_map already seen 605 >= 605 -11816> 2013-04-03 10:41:56.684733 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 update_logger -11815> 2013-04-03 10:41:56.684744 7fcdffcdb700 0 log [INF] : pgmap v29072: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -11814> 2013-04-03 10:41:56.684753 7fcdffcdb700 10 send_log to self -11813> 2013-04-03 10:41:56.684753 7fcdffcdb700 10 log_queue is 1 last_log 138 sent 137 num 1 unsent 1 sending 1 -11812> 2013-04-03 10:41:56.684755 7fcdffcdb700 10 will send 2013-04-03 10:41:56.684752 mon.0 10.252.24.25:6789/0 138 : [INF] pgmap v29072: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -11811> 2013-04-03 10:41:56.684762 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74800 -11810> 2013-04-03 10:41:56.684813 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos -11809> 2013-04-03 10:41:56.684834 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 -11808> 2013-04-03 10:41:56.684853 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11807> 2013-04-03 10:41:56.684878 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74800 con 0x1bac160 -11806> 2013-04-03 10:41:56.684900 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos -11805> 2013-04-03 10:41:56.684916 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 -11804> 2013-04-03 10:41:56.684925 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -11803> 2013-04-03 10:41:56.684938 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 preprocess_log log(1 entries) v1 from mon.0 -11802> 2013-04-03 10:41:56.684963 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -11801> 2013-04-03 10:41:56.684976 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 prepare_log log(1 entries) v1 from mon.0 -11800> 2013-04-03 10:41:56.684988 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 logging 2013-04-03 10:41:56.684752 mon.0 10.252.24.25:6789/0 138 : [INF] pgmap v29072: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -11799> 2013-04-03 10:41:56.695936 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113683 ==== paxos(lease_ack lc 427427 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (405332438 0 0) 0x2c4dc80 con 0x208a840 -11798> 2013-04-03 10:41:56.696019 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos -11797> 2013-04-03 10:41:56.696040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 update_from_paxos version 394069 summary v 394069 -11796> 2013-04-03 10:41:56.696058 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11795> 2013-04-03 10:41:56.936442 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113684 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 127 v187) v2) to leader v1 ==== 560+0+0 (1049301685 0 0) 0x2a832c0 con 0x208a840 -11794> 2013-04-03 10:41:56.936456 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11793> 2013-04-03 10:41:56.936465 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83840 from 10.252.24.88:6789/0 -11792> 2013-04-03 10:41:56.936499 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 127 v187) v2 from mds.0 10.252.24.25:6802/3144 -11791> 2013-04-03 10:41:56.936511 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 127 v187) v2 tid 17634) v2 -- ?+0 0x24f8d80 con 0x208a840 -11790> 2013-04-03 10:41:57.388963 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 127 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21783 v187) v2 ==== 289+0+0 (1920952644 0 0) 0x2cc4840 con 0x200d420 -11789> 2013-04-03 10:41:57.389041 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21783 v187) v2 from mds.? 10.252.24.88:6800/17213 -11788> 2013-04-03 10:41:57.389055 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21783 v187) v2 -- ?+0 0x2a832c0 con 0x200d420 -11787> 2013-04-03 10:41:57.684065 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394069 encode_pending v394070 -11786> 2013-04-03 10:41:57.684092 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427427) queue_proposal bl 443 bytes; ctx = 0x2c3d5e0 -11785> 2013-04-03 10:41:57.684099 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427427) propose_queued 427428 443 bytes -11784> 2013-04-03 10:41:57.685225 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427427 fc 0 pn 17600 opn 0) v3 -- ?+0 0x1f91c80 -11783> 2013-04-03 10:41:57.712326 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113685 ==== paxos(accept lc 427427 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2453904907 0 0) 0x1f91c80 con 0x208a840 -11782> 2013-04-03 10:41:57.712668 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427428 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4dc80 -11781> 2013-04-03 10:41:57.712696 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427428 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2c4d780 -11780> 2013-04-03 10:41:57.712738 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos -11779> 2013-04-03 10:41:57.712776 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos version 394070 summary v 394069 -11778> 2013-04-03 10:41:57.712791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos latest full 393798 -11777> 2013-04-03 10:41:57.712809 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos applying incremental log 394070 2013-04-03 10:41:56.684752 mon.0 10.252.24.25:6789/0 138 : [INF] pgmap v29072: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -11776> 2013-04-03 10:41:57.712858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 check_subs -11775> 2013-04-03 10:41:57.712889 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 create_pending v 394071 -11774> 2013-04-03 10:41:57.712913 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 _updated_log for mon.0 10.252.24.25:6789/0 -11773> 2013-04-03 10:41:57.712926 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 138) v1 -- ?+0 0x2224700 con 0x1bac160 -11772> 2013-04-03 10:41:57.712983 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos -11771> 2013-04-03 10:41:57.712999 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos version 394070 summary v 394070 -11770> 2013-04-03 10:41:57.713018 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11769> 2013-04-03 10:41:57.713039 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 138) v1 ==== 0+0+0 (0 0 0) 0x2224700 con 0x1bac160 -11768> 2013-04-03 10:41:57.713047 7fcdffcdb700 10 handle_log_ack log(last 138) v1 -11767> 2013-04-03 10:41:57.713049 7fcdffcdb700 10 logged 2013-04-03 10:41:56.684752 mon.0 10.252.24.25:6789/0 138 : [INF] pgmap v29072: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -11766> 2013-04-03 10:41:57.737117 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113686 ==== paxos(lease_ack lc 427428 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (88481888 0 0) 0x2c4d780 con 0x208a840 -11765> 2013-04-03 10:41:57.737154 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos -11764> 2013-04-03 10:41:57.737171 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos version 394070 summary v 394070 -11763> 2013-04-03 10:41:57.737186 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11762> 2013-04-03 10:41:58.248639 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113687 ==== forward(pg_stats(1 pgs tid 21 v 605) v1) to leader v1 ==== 757+0+0 (3946052865 0 0) 0x2efa8c0 con 0x208a840 -11761> 2013-04-03 10:41:58.248654 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -11760> 2013-04-03 10:41:58.248663 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -11759> 2013-04-03 10:41:58.248696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 preprocess_query pg_stats(1 pgs tid 21 v 605) v1 from osd.2 10.252.24.30:6801/5656 -11758> 2013-04-03 10:41:58.248718 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 prepare_update pg_stats(1 pgs tid 21 v 605) v1 from osd.2 10.252.24.30:6801/5656 -11757> 2013-04-03 10:41:58.248724 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 prepare_pg_stats pg_stats(1 pgs tid 21 v 605) v1 from osd.2 -11756> 2013-04-03 10:41:58.248731 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -11755> 2013-04-03 10:41:58.358474 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 check_down_pgs -11754> 2013-04-03 10:41:58.358510 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29072 v29072: 240 pgs: 240 active+clean; 254 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1172B/s wr, 0op/s -11753> 2013-04-03 10:41:58.358545 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -11752> 2013-04-03 10:41:58.358571 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -11751> 2013-04-03 10:41:58.358608 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -11750> 2013-04-03 10:41:58.358640 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos -11749> 2013-04-03 10:41:58.358659 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 update_from_paxos version 394070 summary v 394070 -11748> 2013-04-03 10:41:58.358669 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394070 log -11747> 2013-04-03 10:41:58.358686 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11746> 2013-04-03 10:41:58.358705 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -11745> 2013-04-03 10:41:58.597490 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) -11744> 2013-04-03 10:41:58.597520 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader -11743> 2013-04-03 10:41:58.712648 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29072 encode_pending v 29073 -11742> 2013-04-03 10:41:58.712690 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427428) queue_proposal bl 654 bytes; ctx = 0x2c3d720 -11741> 2013-04-03 10:41:58.712700 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427428) propose_queued 427429 654 bytes -11740> 2013-04-03 10:41:58.713856 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427428 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116780 -11739> 2013-04-03 10:41:58.745095 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113688 ==== paxos(accept lc 427428 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2800580221 0 0) 0x2116780 con 0x208a840 -11738> 2013-04-03 10:41:58.745473 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427429 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2c4d780 -11737> 2013-04-03 10:41:58.745498 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427429 fc 427382 pn 0 opn 0) v3 -- ?+0 0x1f91c80 -11736> 2013-04-03 10:41:58.745586 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29072 update_from_paxos applying incremental 29073 -11735> 2013-04-03 10:41:58.745606 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 v29073: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 7078B/s wr, 0op/s -11734> 2013-04-03 10:41:58.745624 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 send_pg_creates to 0 pgs -11733> 2013-04-03 10:41:58.745626 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 update_logger -11732> 2013-04-03 10:41:58.745636 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 create_pending v 29074 -11731> 2013-04-03 10:41:58.745644 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29073 _updated_stats for osd.2 10.252.24.30:6801/5656 -11730> 2013-04-03 10:41:58.745652 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 21) v1 tid 17635) v2 -- ?+0 0x24f8900 con 0x208a840 -11729> 2013-04-03 10:41:58.745662 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 check_osd_map already seen 605 >= 605 -11728> 2013-04-03 10:41:58.745665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 update_logger -11727> 2013-04-03 10:41:58.745675 7fcdffcdb700 0 log [INF] : pgmap v29073: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 7078B/s wr, 0op/s -11726> 2013-04-03 10:41:58.745677 7fcdffcdb700 10 send_log to self -11725> 2013-04-03 10:41:58.745678 7fcdffcdb700 10 log_queue is 1 last_log 139 sent 138 num 1 unsent 1 sending 1 -11724> 2013-04-03 10:41:58.745679 7fcdffcdb700 10 will send 2013-04-03 10:41:58.745676 mon.0 10.252.24.25:6789/0 139 : [INF] pgmap v29073: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 7078B/s wr, 0op/s -11723> 2013-04-03 10:41:58.745686 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74800 -11722> 2013-04-03 10:41:58.745737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 update_from_paxos -11721> 2013-04-03 10:41:58.745756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 update_from_paxos version 394070 summary v 394070 -11720> 2013-04-03 10:41:58.745775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11719> 2013-04-03 10:41:58.745796 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74800 con 0x1bac160 -11718> 2013-04-03 10:41:58.745830 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 update_from_paxos -11717> 2013-04-03 10:41:58.745846 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 update_from_paxos version 394070 summary v 394070 -11716> 2013-04-03 10:41:58.745855 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -11715> 2013-04-03 10:41:58.745867 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 preprocess_log log(1 entries) v1 from mon.0 -11714> 2013-04-03 10:41:58.745891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -11713> 2013-04-03 10:41:58.745903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 prepare_log log(1 entries) v1 from mon.0 -11712> 2013-04-03 10:41:58.745914 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 logging 2013-04-03 10:41:58.745676 mon.0 10.252.24.25:6789/0 139 : [INF] pgmap v29073: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 7078B/s wr, 0op/s -11711> 2013-04-03 10:41:58.756746 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113689 ==== paxos(lease_ack lc 427429 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (138000868 0 0) 0x1f91c80 con 0x208a840 -11710> 2013-04-03 10:41:58.756780 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 update_from_paxos -11709> 2013-04-03 10:41:58.756796 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394070 update_from_paxos version 394070 summary v 394070 -11708> 2013-04-03 10:41:58.756812 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11707> 2013-04-03 10:41:59.590381 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 128 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140413 v187) v2 ==== 289+0+0 (1464457588 0 0) 0x2efa340 con 0x2118420 -11706> 2013-04-03 10:41:59.590458 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140413 v187) v2 from mds.? 10.252.58.31:6800/10908 -11705> 2013-04-03 10:41:59.590472 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140413 v187) v2 -- ?+0 0x2efa8c0 con 0x2118420 -11704> 2013-04-03 10:41:59.745312 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394070 encode_pending v394071 -11703> 2013-04-03 10:41:59.745341 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427429) queue_proposal bl 443 bytes; ctx = 0x2c3d610 -11702> 2013-04-03 10:41:59.745348 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427429) propose_queued 427430 443 bytes -11701> 2013-04-03 10:41:59.746480 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427429 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116000 -11700> 2013-04-03 10:41:59.772743 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113690 ==== paxos(accept lc 427429 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1852952037 0 0) 0x2116000 con 0x208a840 -11699> 2013-04-03 10:41:59.773609 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427430 fc 0 pn 17600 opn 0) v3 -- ?+0 0x1f91c80 -11698> 2013-04-03 10:41:59.773640 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427430 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2116780 -11697> 2013-04-03 10:41:59.773678 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos -11696> 2013-04-03 10:41:59.773705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394070 -11695> 2013-04-03 10:41:59.773724 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos latest full 393798 -11694> 2013-04-03 10:41:59.773742 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394071 update_from_paxos applying incremental log 394071 2013-04-03 10:41:58.745676 mon.0 10.252.24.25:6789/0 139 : [INF] pgmap v29073: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 7078B/s wr, 0op/s -11693> 2013-04-03 10:41:59.773791 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 check_subs -11692> 2013-04-03 10:41:59.773822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 create_pending v 394072 -11691> 2013-04-03 10:41:59.773844 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394071 _updated_log for mon.0 10.252.24.25:6789/0 -11690> 2013-04-03 10:41:59.773856 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 139) v1 -- ?+0 0x2e8be00 con 0x1bac160 -11689> 2013-04-03 10:41:59.773908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos -11688> 2013-04-03 10:41:59.773924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394071 -11687> 2013-04-03 10:41:59.773942 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11686> 2013-04-03 10:41:59.773964 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 139) v1 ==== 0+0+0 (0 0 0) 0x2e8be00 con 0x1bac160 -11685> 2013-04-03 10:41:59.773972 7fcdffcdb700 10 handle_log_ack log(last 139) v1 -11684> 2013-04-03 10:41:59.773974 7fcdffcdb700 10 logged 2013-04-03 10:41:58.745676 mon.0 10.252.24.25:6789/0 139 : [INF] pgmap v29073: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 7078B/s wr, 0op/s -11683> 2013-04-03 10:41:59.784578 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113691 ==== paxos(lease_ack lc 427430 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2077187546 0 0) 0x2116780 con 0x208a840 -11682> 2013-04-03 10:41:59.784614 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos -11681> 2013-04-03 10:41:59.784631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394071 -11680> 2013-04-03 10:41:59.784647 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11679> 2013-04-03 10:42:00.936603 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113692 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 128 v187) v2) to leader v1 ==== 560+0+0 (3208417915 0 0) 0x2a84b80 con 0x208a840 -11678> 2013-04-03 10:42:00.936624 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11677> 2013-04-03 10:42:00.936633 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a512c0 from 10.252.24.88:6789/0 -11676> 2013-04-03 10:42:00.936688 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 128 v187) v2 from mds.0 10.252.24.25:6802/3144 -11675> 2013-04-03 10:42:00.936699 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 128 v187) v2 tid 17636) v2 -- ?+0 0x24f8d80 con 0x208a840 -11674> 2013-04-03 10:42:01.389065 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 128 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21784 v187) v2 ==== 289+0+0 (2230310958 0 0) 0x273a580 con 0x200d420 -11673> 2013-04-03 10:42:01.389141 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21784 v187) v2 from mds.? 10.252.24.88:6800/17213 -11672> 2013-04-03 10:42:01.389154 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21784 v187) v2 -- ?+0 0x2a84b80 con 0x200d420 -11671> 2013-04-03 10:42:02.773713 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427430 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2116280 -11670> 2013-04-03 10:42:02.774624 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113693 ==== paxos(lease_ack lc 427430 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2379510561 0 0) 0x2116280 con 0x208a840 -11669> 2013-04-03 10:42:02.774697 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos -11668> 2013-04-03 10:42:02.774717 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394071 -11667> 2013-04-03 10:42:02.774734 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11666> 2013-04-03 10:42:03.249544 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113694 ==== forward(pg_stats(1 pgs tid 22 v 605) v1) to leader v1 ==== 757+0+0 (32525352 0 0) 0x2efa340 con 0x208a840 -11665> 2013-04-03 10:42:03.249558 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -11664> 2013-04-03 10:42:03.249568 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -11663> 2013-04-03 10:42:03.249599 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 preprocess_query pg_stats(1 pgs tid 22 v 605) v1 from osd.2 10.252.24.30:6801/5656 -11662> 2013-04-03 10:42:03.249620 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 prepare_update pg_stats(1 pgs tid 22 v 605) v1 from osd.2 10.252.24.30:6801/5656 -11661> 2013-04-03 10:42:03.249626 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 prepare_pg_stats pg_stats(1 pgs tid 22 v 605) v1 from osd.2 -11660> 2013-04-03 10:42:03.249632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29073 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -11659> 2013-04-03 10:42:03.299816 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29073 encode_pending v 29074 -11658> 2013-04-03 10:42:03.299852 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427430) queue_proposal bl 654 bytes; ctx = 0x2c3d730 -11657> 2013-04-03 10:42:03.299861 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427430) propose_queued 427431 654 bytes -11656> 2013-04-03 10:42:03.301331 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427430 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116500 -11655> 2013-04-03 10:42:03.358855 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -11654> 2013-04-03 10:42:03.358894 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -11653> 2013-04-03 10:42:03.358917 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos -11652> 2013-04-03 10:42:03.358936 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394071 -11651> 2013-04-03 10:42:03.358945 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394071 log -11650> 2013-04-03 10:42:03.358962 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11649> 2013-04-03 10:42:03.358981 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -11648> 2013-04-03 10:42:03.364721 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113695 ==== paxos(accept lc 427430 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (73784524 0 0) 0x2116500 con 0x208a840 -11647> 2013-04-03 10:42:03.365078 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427431 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116280 -11646> 2013-04-03 10:42:03.365108 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427431 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2116780 -11645> 2013-04-03 10:42:03.365191 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29073 update_from_paxos applying incremental 29074 -11644> 2013-04-03 10:42:03.365211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 v29074: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 11097B/s wr, 0op/s -11643> 2013-04-03 10:42:03.365230 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 send_pg_creates to 0 pgs -11642> 2013-04-03 10:42:03.365234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 update_logger -11641> 2013-04-03 10:42:03.365244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 create_pending v 29075 -11640> 2013-04-03 10:42:03.365253 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29074 _updated_stats for osd.2 10.252.24.30:6801/5656 -11639> 2013-04-03 10:42:03.365259 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 22) v1 tid 17637) v2 -- ?+0 0x24f8900 con 0x208a840 -11638> 2013-04-03 10:42:03.365270 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 check_osd_map already seen 605 >= 605 -11637> 2013-04-03 10:42:03.365272 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 update_logger -11636> 2013-04-03 10:42:03.365283 7fcdffcdb700 0 log [INF] : pgmap v29074: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 11097B/s wr, 0op/s -11635> 2013-04-03 10:42:03.365284 7fcdffcdb700 10 send_log to self -11634> 2013-04-03 10:42:03.365285 7fcdffcdb700 10 log_queue is 1 last_log 140 sent 139 num 1 unsent 1 sending 1 -11633> 2013-04-03 10:42:03.365287 7fcdffcdb700 10 will send 2013-04-03 10:42:03.365284 mon.0 10.252.24.25:6789/0 140 : [INF] pgmap v29074: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 11097B/s wr, 0op/s -11632> 2013-04-03 10:42:03.365295 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74800 -11631> 2013-04-03 10:42:03.365343 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos -11630> 2013-04-03 10:42:03.365360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394071 -11629> 2013-04-03 10:42:03.365377 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11628> 2013-04-03 10:42:03.365398 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74800 con 0x1bac160 -11627> 2013-04-03 10:42:03.365419 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos -11626> 2013-04-03 10:42:03.365436 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394071 -11625> 2013-04-03 10:42:03.365445 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -11624> 2013-04-03 10:42:03.365799 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 preprocess_log log(1 entries) v1 from mon.0 -11623> 2013-04-03 10:42:03.365839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -11622> 2013-04-03 10:42:03.365854 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 prepare_log log(1 entries) v1 from mon.0 -11621> 2013-04-03 10:42:03.365865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 logging 2013-04-03 10:42:03.365284 mon.0 10.252.24.25:6789/0 140 : [INF] pgmap v29074: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 11097B/s wr, 0op/s -11620> 2013-04-03 10:42:03.377895 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113696 ==== paxos(lease_ack lc 427431 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3649951181 0 0) 0x2116780 con 0x208a840 -11619> 2013-04-03 10:42:03.377932 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos -11618> 2013-04-03 10:42:03.377948 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394071 update_from_paxos version 394071 summary v 394071 -11617> 2013-04-03 10:42:03.377963 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11616> 2013-04-03 10:42:03.590473 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 129 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140414 v187) v2 ==== 289+0+0 (855251187 0 0) 0x2efa080 con 0x2118420 -11615> 2013-04-03 10:42:03.590521 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140414 v187) v2 from mds.? 10.252.58.31:6800/10908 -11614> 2013-04-03 10:42:03.590531 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140414 v187) v2 -- ?+0 0x2efa340 con 0x2118420 -11613> 2013-04-03 10:42:04.364927 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394071 encode_pending v394072 -11612> 2013-04-03 10:42:04.364955 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427431) queue_proposal bl 444 bytes; ctx = 0x2c3d5d0 -11611> 2013-04-03 10:42:04.364962 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427431) propose_queued 427432 444 bytes -11610> 2013-04-03 10:42:04.366162 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427431 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116a00 -11609> 2013-04-03 10:42:04.409707 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113697 ==== paxos(accept lc 427431 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (52283917 0 0) 0x2116a00 con 0x208a840 -11608> 2013-04-03 10:42:04.410029 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427432 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116780 -11607> 2013-04-03 10:42:04.410062 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427432 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2116500 -11606> 2013-04-03 10:42:04.410102 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos -11605> 2013-04-03 10:42:04.410129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394071 -11604> 2013-04-03 10:42:04.410143 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos latest full 393798 -11603> 2013-04-03 10:42:04.410164 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394072 update_from_paxos applying incremental log 394072 2013-04-03 10:42:03.365284 mon.0 10.252.24.25:6789/0 140 : [INF] pgmap v29074: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 11097B/s wr, 0op/s -11602> 2013-04-03 10:42:04.410209 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 check_subs -11601> 2013-04-03 10:42:04.410240 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 create_pending v 394073 -11600> 2013-04-03 10:42:04.410262 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394072 _updated_log for mon.0 10.252.24.25:6789/0 -11599> 2013-04-03 10:42:04.410275 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 140) v1 -- ?+0 0x2e8bc40 con 0x1bac160 -11598> 2013-04-03 10:42:04.410327 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos -11597> 2013-04-03 10:42:04.410342 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 -11596> 2013-04-03 10:42:04.410360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11595> 2013-04-03 10:42:04.410382 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 140) v1 ==== 0+0+0 (0 0 0) 0x2e8bc40 con 0x1bac160 -11594> 2013-04-03 10:42:04.410390 7fcdffcdb700 10 handle_log_ack log(last 140) v1 -11593> 2013-04-03 10:42:04.410392 7fcdffcdb700 10 logged 2013-04-03 10:42:03.365284 mon.0 10.252.24.25:6789/0 140 : [INF] pgmap v29074: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 11097B/s wr, 0op/s -11592> 2013-04-03 10:42:04.422965 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113698 ==== paxos(lease_ack lc 427432 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (4174543082 0 0) 0x2116500 con 0x208a840 -11591> 2013-04-03 10:42:04.423000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos -11590> 2013-04-03 10:42:04.423016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 -11589> 2013-04-03 10:42:04.423031 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11588> 2013-04-03 10:42:04.936741 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113699 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 129 v187) v2) to leader v1 ==== 560+0+0 (1648989998 0 0) 0x2a52b80 con 0x208a840 -11587> 2013-04-03 10:42:04.936763 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11586> 2013-04-03 10:42:04.936772 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52080 from 10.252.24.88:6789/0 -11585> 2013-04-03 10:42:04.936822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 129 v187) v2 from mds.0 10.252.24.25:6802/3144 -11584> 2013-04-03 10:42:04.936837 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 129 v187) v2 tid 17638) v2 -- ?+0 0x24f8d80 con 0x208a840 -11583> 2013-04-03 10:42:05.389151 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 129 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21785 v187) v2 ==== 289+0+0 (2810275667 0 0) 0x273b340 con 0x200d420 -11582> 2013-04-03 10:42:05.389227 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21785 v187) v2 from mds.? 10.252.24.88:6800/17213 -11581> 2013-04-03 10:42:05.389240 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21785 v187) v2 -- ?+0 0x2a52b80 con 0x200d420 -11580> 2013-04-03 10:42:07.410146 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427432 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2df2c80 -11579> 2013-04-03 10:42:07.410910 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113700 ==== paxos(lease_ack lc 427432 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3106567010 0 0) 0x2df2c80 con 0x208a840 -11578> 2013-04-03 10:42:07.410985 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos -11577> 2013-04-03 10:42:07.411005 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 -11576> 2013-04-03 10:42:07.411022 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11575> 2013-04-03 10:42:07.590588 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 130 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140415 v187) v2 ==== 289+0+0 (294954894 0 0) 0x2ef9dc0 con 0x2118420 -11574> 2013-04-03 10:42:07.590641 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140415 v187) v2 from mds.? 10.252.58.31:6800/10908 -11573> 2013-04-03 10:42:07.590652 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140415 v187) v2 -- ?+0 0x273b340 con 0x2118420 -11572> 2013-04-03 10:42:08.359101 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29074 check_down_pgs -11571> 2013-04-03 10:42:08.359130 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29074 v29074: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 11097B/s wr, 0op/s -11570> 2013-04-03 10:42:08.359158 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -11569> 2013-04-03 10:42:08.359187 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -11568> 2013-04-03 10:42:08.359223 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -11567> 2013-04-03 10:42:08.359254 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos -11566> 2013-04-03 10:42:08.359273 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 -11565> 2013-04-03 10:42:08.359282 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394072 log -11564> 2013-04-03 10:42:08.359299 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11563> 2013-04-03 10:42:08.359318 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -11562> 2013-04-03 10:42:08.936906 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113701 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 130 v187) v2) to leader v1 ==== 560+0+0 (3787155279 0 0) 0x2efa080 con 0x208a840 -11561> 2013-04-03 10:42:08.936928 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11560> 2013-04-03 10:42:08.936937 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a51000 from 10.252.24.88:6789/0 -11559> 2013-04-03 10:42:08.936989 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 130 v187) v2 from mds.0 10.252.24.25:6802/3144 -11558> 2013-04-03 10:42:08.937000 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 130 v187) v2 tid 17639) v2 -- ?+0 0x24f8900 con 0x208a840 -11557> 2013-04-03 10:42:09.389269 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 130 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21786 v187) v2 ==== 289+0+0 (3258099412 0 0) 0x273a2c0 con 0x200d420 -11556> 2013-04-03 10:42:09.389347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21786 v187) v2 from mds.? 10.252.24.88:6800/17213 -11555> 2013-04-03 10:42:09.389360 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21786 v187) v2 -- ?+0 0x2efa080 con 0x200d420 -11554> 2013-04-03 10:42:10.410215 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427432 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2df2a00 -11553> 2013-04-03 10:42:10.411014 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113702 ==== paxos(lease_ack lc 427432 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2998048961 0 0) 0x2df2a00 con 0x208a840 -11552> 2013-04-03 10:42:10.411087 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos -11551> 2013-04-03 10:42:10.411108 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 -11550> 2013-04-03 10:42:10.411143 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11549> 2013-04-03 10:42:11.590748 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 131 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140416 v187) v2 ==== 289+0+0 (3102340891 0 0) 0x2ef9b00 con 0x2118420 -11548> 2013-04-03 10:42:11.590825 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140416 v187) v2 from mds.? 10.252.58.31:6800/10908 -11547> 2013-04-03 10:42:11.590839 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140416 v187) v2 -- ?+0 0x273a2c0 con 0x2118420 -11546> 2013-04-03 10:42:11.607253 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113703 ==== forward(pg_stats(1 pgs tid 42 v 605) v1) to leader v1 ==== 753+0+0 (644078511 0 0) 0x2ef9dc0 con 0x208a840 -11545> 2013-04-03 10:42:11.607266 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -11544> 2013-04-03 10:42:11.607275 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -11543> 2013-04-03 10:42:11.607307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 preprocess_query pg_stats(1 pgs tid 42 v 605) v1 from osd.1 10.252.24.25:6805/3572 -11542> 2013-04-03 10:42:11.607331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 prepare_update pg_stats(1 pgs tid 42 v 605) v1 from osd.1 10.252.24.25:6805/3572 -11541> 2013-04-03 10:42:11.607337 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 prepare_pg_stats pg_stats(1 pgs tid 42 v 605) v1 from osd.1 -11540> 2013-04-03 10:42:11.607343 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29074 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -11539> 2013-04-03 10:42:11.657528 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29074 encode_pending v 29075 -11538> 2013-04-03 10:42:11.657561 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427432) queue_proposal bl 650 bytes; ctx = 0x2c3d700 -11537> 2013-04-03 10:42:11.657571 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427432) propose_queued 427433 650 bytes -11536> 2013-04-03 10:42:11.659487 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427432 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2780 -11535> 2013-04-03 10:42:11.690945 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113704 ==== paxos(accept lc 427432 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1210025540 0 0) 0x2df2780 con 0x208a840 -11534> 2013-04-03 10:42:11.691302 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427433 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2a00 -11533> 2013-04-03 10:42:11.691331 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427433 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2df2c80 -11532> 2013-04-03 10:42:11.691398 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29074 update_from_paxos applying incremental 29075 -11531> 2013-04-03 10:42:11.691417 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 v29075: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 632B/s wr, 0op/s -11530> 2013-04-03 10:42:11.691438 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 send_pg_creates to 0 pgs -11529> 2013-04-03 10:42:11.691443 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 update_logger -11528> 2013-04-03 10:42:11.691453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 create_pending v 29076 -11527> 2013-04-03 10:42:11.691461 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29075 _updated_stats for osd.1 10.252.24.25:6805/3572 -11526> 2013-04-03 10:42:11.691468 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 42) v1 tid 17640) v2 -- ?+0 0x24f8d80 con 0x208a840 -11525> 2013-04-03 10:42:11.691478 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 check_osd_map already seen 605 >= 605 -11524> 2013-04-03 10:42:11.691481 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 update_logger -11523> 2013-04-03 10:42:11.691492 7fcdffcdb700 0 log [INF] : pgmap v29075: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 632B/s wr, 0op/s -11522> 2013-04-03 10:42:11.691494 7fcdffcdb700 10 send_log to self -11521> 2013-04-03 10:42:11.691494 7fcdffcdb700 10 log_queue is 1 last_log 141 sent 140 num 1 unsent 1 sending 1 -11520> 2013-04-03 10:42:11.691496 7fcdffcdb700 10 will send 2013-04-03 10:42:11.691493 mon.0 10.252.24.25:6789/0 141 : [INF] pgmap v29075: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 632B/s wr, 0op/s -11519> 2013-04-03 10:42:11.691503 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74800 -11518> 2013-04-03 10:42:11.691647 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos -11517> 2013-04-03 10:42:11.691669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 -11516> 2013-04-03 10:42:11.691688 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11515> 2013-04-03 10:42:11.691709 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74800 con 0x1bac160 -11514> 2013-04-03 10:42:11.691732 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos -11513> 2013-04-03 10:42:11.691746 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 -11512> 2013-04-03 10:42:11.691755 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -11511> 2013-04-03 10:42:11.691769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 preprocess_log log(1 entries) v1 from mon.0 -11510> 2013-04-03 10:42:11.691792 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -11509> 2013-04-03 10:42:11.691804 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 prepare_log log(1 entries) v1 from mon.0 -11508> 2013-04-03 10:42:11.691813 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 logging 2013-04-03 10:42:11.691493 mon.0 10.252.24.25:6789/0 141 : [INF] pgmap v29075: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 632B/s wr, 0op/s -11507> 2013-04-03 10:42:11.702912 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113705 ==== paxos(lease_ack lc 427433 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3763837724 0 0) 0x2df2c80 con 0x208a840 -11506> 2013-04-03 10:42:11.702948 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos -11505> 2013-04-03 10:42:11.702965 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394072 update_from_paxos version 394072 summary v 394072 -11504> 2013-04-03 10:42:11.702979 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11503> 2013-04-03 10:42:11.772235 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113706 ==== forward(pg_stats(1 pgs tid 30453 v 605) v1) to leader v1 ==== 757+0+0 (3265116277 0 0) 0x1bae8c0 con 0x208a840 -11502> 2013-04-03 10:42:11.772247 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -11501> 2013-04-03 10:42:11.772254 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -11500> 2013-04-03 10:42:11.772281 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 preprocess_query pg_stats(1 pgs tid 30453 v 605) v1 from osd.3 10.252.24.88:6801/17538 -11499> 2013-04-03 10:42:11.772301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 prepare_update pg_stats(1 pgs tid 30453 v 605) v1 from osd.3 10.252.24.88:6801/17538 -11498> 2013-04-03 10:42:11.772306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 prepare_pg_stats pg_stats(1 pgs tid 30453 v 605) v1 from osd.3 -11497> 2013-04-03 10:42:11.772310 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29075 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -11496> 2013-04-03 10:42:12.691148 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394072 encode_pending v394073 -11495> 2013-04-03 10:42:12.691176 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427433) queue_proposal bl 442 bytes; ctx = 0x2c3d880 -11494> 2013-04-03 10:42:12.691183 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427433) propose_queued 427434 442 bytes -11493> 2013-04-03 10:42:12.692392 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427433 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2500 -11492> 2013-04-03 10:42:12.692683 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29075 encode_pending v 29076 -11491> 2013-04-03 10:42:12.692717 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427382..427433) queue_proposal bl 654 bytes; ctx = 0x2c3d870 -11490> 2013-04-03 10:42:12.692724 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427382..427433) propose_new_value not active; proposal queued -11489> 2013-04-03 10:42:12.718793 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113707 ==== paxos(accept lc 427433 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1212663745 0 0) 0x2df2500 con 0x208a840 -11488> 2013-04-03 10:42:12.719098 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427434 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2c80 -11487> 2013-04-03 10:42:12.719129 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427434 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2df2780 -11486> 2013-04-03 10:42:12.719167 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos -11485> 2013-04-03 10:42:12.719196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos version 394073 summary v 394072 -11484> 2013-04-03 10:42:12.719212 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos latest full 393798 -11483> 2013-04-03 10:42:12.719231 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394073 update_from_paxos applying incremental log 394073 2013-04-03 10:42:11.691493 mon.0 10.252.24.25:6789/0 141 : [INF] pgmap v29075: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 632B/s wr, 0op/s -11482> 2013-04-03 10:42:12.719407 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 check_subs -11481> 2013-04-03 10:42:12.719441 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 create_pending v 394074 -11480> 2013-04-03 10:42:12.719463 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394073 _updated_log for mon.0 10.252.24.25:6789/0 -11479> 2013-04-03 10:42:12.719476 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 141) v1 -- ?+0 0x2e8b8c0 con 0x1bac160 -11478> 2013-04-03 10:42:12.719500 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427434) propose_queued 427435 654 bytes -11477> 2013-04-03 10:42:12.719686 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427434 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116500 -11476> 2013-04-03 10:42:12.719719 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 141) v1 ==== 0+0+0 (0 0 0) 0x2e8b8c0 con 0x1bac160 -11475> 2013-04-03 10:42:12.719739 7fcdffcdb700 10 handle_log_ack log(last 141) v1 -11474> 2013-04-03 10:42:12.719741 7fcdffcdb700 10 logged 2013-04-03 10:42:11.691493 mon.0 10.252.24.25:6789/0 141 : [INF] pgmap v29075: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 632B/s wr, 0op/s -11473> 2013-04-03 10:42:12.730712 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113708 ==== paxos(lease_ack lc 427434 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (384857037 0 0) 0x2116500 con 0x208a840 -11472> 2013-04-03 10:42:12.743935 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113709 ==== paxos(accept lc 427434 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (949502881 0 0) 0x2df2780 con 0x208a840 -11471> 2013-04-03 10:42:12.744235 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427435 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116500 -11470> 2013-04-03 10:42:12.744266 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427435 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2df2500 -11469> 2013-04-03 10:42:12.744797 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29075 update_from_paxos applying incremental 29076 -11468> 2013-04-03 10:42:12.744820 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 v29076: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -11467> 2013-04-03 10:42:12.744843 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 send_pg_creates to 0 pgs -11466> 2013-04-03 10:42:12.744848 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 update_logger -11465> 2013-04-03 10:42:12.744859 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 create_pending v 29077 -11464> 2013-04-03 10:42:12.744868 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29076 _updated_stats for osd.3 10.252.24.88:6801/17538 -11463> 2013-04-03 10:42:12.744875 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30453) v1 tid 17641) v2 -- ?+0 0x24f8900 con 0x208a840 -11462> 2013-04-03 10:42:12.744885 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 check_osd_map already seen 605 >= 605 -11461> 2013-04-03 10:42:12.744888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 update_logger -11460> 2013-04-03 10:42:12.744898 7fcdffcdb700 0 log [INF] : pgmap v29076: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -11459> 2013-04-03 10:42:12.744899 7fcdffcdb700 10 send_log to self -11458> 2013-04-03 10:42:12.744900 7fcdffcdb700 10 log_queue is 1 last_log 142 sent 141 num 1 unsent 1 sending 1 -11457> 2013-04-03 10:42:12.744902 7fcdffcdb700 10 will send 2013-04-03 10:42:12.744899 mon.0 10.252.24.25:6789/0 142 : [INF] pgmap v29076: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -11456> 2013-04-03 10:42:12.744908 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73400 -11455> 2013-04-03 10:42:12.744957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos -11454> 2013-04-03 10:42:12.744974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos version 394073 summary v 394073 -11453> 2013-04-03 10:42:12.744992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11452> 2013-04-03 10:42:12.745012 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73400 con 0x1bac160 -11451> 2013-04-03 10:42:12.745033 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos -11450> 2013-04-03 10:42:12.745048 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos version 394073 summary v 394073 -11449> 2013-04-03 10:42:12.745057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -11448> 2013-04-03 10:42:12.745069 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 preprocess_log log(1 entries) v1 from mon.0 -11447> 2013-04-03 10:42:12.745092 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -11446> 2013-04-03 10:42:12.745105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 prepare_log log(1 entries) v1 from mon.0 -11445> 2013-04-03 10:42:12.745116 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 logging 2013-04-03 10:42:12.744899 mon.0 10.252.24.25:6789/0 142 : [INF] pgmap v29076: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -11444> 2013-04-03 10:42:12.769389 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113710 ==== paxos(lease_ack lc 427435 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (4170519173 0 0) 0x2df2500 con 0x208a840 -11443> 2013-04-03 10:42:12.769424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos -11442> 2013-04-03 10:42:12.769440 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos version 394073 summary v 394073 -11441> 2013-04-03 10:42:12.769455 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11440> 2013-04-03 10:42:12.937013 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113711 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 131 v187) v2) to leader v1 ==== 560+0+0 (3056946951 0 0) 0x1baeb80 con 0x208a840 -11439> 2013-04-03 10:42:12.937034 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11438> 2013-04-03 10:42:12.937043 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4340 from 10.252.24.88:6789/0 -11437> 2013-04-03 10:42:12.937093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 131 v187) v2 from mds.0 10.252.24.25:6802/3144 -11436> 2013-04-03 10:42:12.937104 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 131 v187) v2 tid 17642) v2 -- ?+0 0x24f8d80 con 0x208a840 -11435> 2013-04-03 10:42:13.250775 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113712 ==== forward(pg_stats(1 pgs tid 23 v 605) v1) to leader v1 ==== 757+0+0 (348672225 0 0) 0x1bae8c0 con 0x208a840 -11434> 2013-04-03 10:42:13.250789 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -11433> 2013-04-03 10:42:13.250796 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -11432> 2013-04-03 10:42:13.250829 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 preprocess_query pg_stats(1 pgs tid 23 v 605) v1 from osd.2 10.252.24.30:6801/5656 -11431> 2013-04-03 10:42:13.250853 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 prepare_update pg_stats(1 pgs tid 23 v 605) v1 from osd.2 10.252.24.30:6801/5656 -11430> 2013-04-03 10:42:13.250859 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 prepare_pg_stats pg_stats(1 pgs tid 23 v 605) v1 from osd.2 -11429> 2013-04-03 10:42:13.250865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29076 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -11428> 2013-04-03 10:42:13.359469 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29076 check_down_pgs -11427> 2013-04-03 10:42:13.359499 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29076 v29076: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -11426> 2013-04-03 10:42:13.359526 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -11425> 2013-04-03 10:42:13.359551 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -11424> 2013-04-03 10:42:13.359587 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -11423> 2013-04-03 10:42:13.359618 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos -11422> 2013-04-03 10:42:13.359636 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394073 update_from_paxos version 394073 summary v 394073 -11421> 2013-04-03 10:42:13.359645 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394073 log -11420> 2013-04-03 10:42:13.359662 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11419> 2013-04-03 10:42:13.359681 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -11418> 2013-04-03 10:42:13.389372 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 131 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21787 v187) v2 ==== 289+0+0 (3780909481 0 0) 0x2f56b80 con 0x200d420 -11417> 2013-04-03 10:42:13.389423 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21787 v187) v2 from mds.? 10.252.24.88:6800/17213 -11416> 2013-04-03 10:42:13.389434 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21787 v187) v2 -- ?+0 0x1bae8c0 con 0x200d420 -11415> 2013-04-03 10:42:13.744106 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29076 encode_pending v 29077 -11414> 2013-04-03 10:42:13.744146 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427435) queue_proposal bl 654 bytes; ctx = 0x2c3d710 -11413> 2013-04-03 10:42:13.744155 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427435) propose_queued 427436 654 bytes -11412> 2013-04-03 10:42:13.745546 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427435 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2280 -11411> 2013-04-03 10:42:13.745641 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394073 encode_pending v394074 -11410> 2013-04-03 10:42:13.745667 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427382..427435) queue_proposal bl 442 bytes; ctx = 0x2c3d6f0 -11409> 2013-04-03 10:42:13.745674 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427382..427435) propose_new_value not active; proposal queued -11408> 2013-04-03 10:42:13.768179 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113713 ==== paxos(accept lc 427435 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3789470310 0 0) 0x2df2280 con 0x208a840 -11407> 2013-04-03 10:42:13.774684 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427436 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2500 -11406> 2013-04-03 10:42:13.774714 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427436 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2df2780 -11405> 2013-04-03 10:42:13.774807 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29076 update_from_paxos applying incremental 29077 -11404> 2013-04-03 10:42:13.774826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 v29077: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 12268B/s wr, 0op/s -11403> 2013-04-03 10:42:13.774844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 send_pg_creates to 0 pgs -11402> 2013-04-03 10:42:13.774847 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 update_logger -11401> 2013-04-03 10:42:13.774856 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 create_pending v 29078 -11400> 2013-04-03 10:42:13.774864 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29077 _updated_stats for osd.2 10.252.24.30:6801/5656 -11399> 2013-04-03 10:42:13.774871 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 23) v1 tid 17643) v2 -- ?+0 0x24f8900 con 0x208a840 -11398> 2013-04-03 10:42:13.774881 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 check_osd_map already seen 605 >= 605 -11397> 2013-04-03 10:42:13.774885 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 update_logger -11396> 2013-04-03 10:42:13.774895 7fcdffcdb700 0 log [INF] : pgmap v29077: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 12268B/s wr, 0op/s -11395> 2013-04-03 10:42:13.774896 7fcdffcdb700 10 send_log to self -11394> 2013-04-03 10:42:13.774897 7fcdffcdb700 10 log_queue is 2 last_log 143 sent 142 num 2 unsent 1 sending 1 -11393> 2013-04-03 10:42:13.774899 7fcdffcdb700 10 will send 2013-04-03 10:42:12.744899 mon.0 10.252.24.25:6789/0 142 : [INF] pgmap v29076: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -11392> 2013-04-03 10:42:13.774912 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73600 -11391> 2013-04-03 10:42:13.774941 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427436) propose_queued 427437 442 bytes -11390> 2013-04-03 10:42:13.775150 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427436 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116a00 -11389> 2013-04-03 10:42:13.775185 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73600 con 0x1bac160 -11388> 2013-04-03 10:42:13.793377 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113714 ==== paxos(lease_ack lc 427436 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1148512059 0 0) 0x2116a00 con 0x208a840 -11387> 2013-04-03 10:42:13.805192 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113715 ==== paxos(accept lc 427436 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (931910969 0 0) 0x2df2780 con 0x208a840 -11386> 2013-04-03 10:42:13.806223 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427437 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116a00 -11385> 2013-04-03 10:42:13.806255 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427437 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2df2280 -11384> 2013-04-03 10:42:13.806288 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos -11383> 2013-04-03 10:42:13.806312 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394073 -11382> 2013-04-03 10:42:13.806326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos latest full 393798 -11381> 2013-04-03 10:42:13.806342 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394074 update_from_paxos applying incremental log 394074 2013-04-03 10:42:12.744899 mon.0 10.252.24.25:6789/0 142 : [INF] pgmap v29076: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -11380> 2013-04-03 10:42:13.806386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 check_subs -11379> 2013-04-03 10:42:13.806416 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 create_pending v 394075 -11378> 2013-04-03 10:42:13.806438 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394074 _updated_log for mon.0 10.252.24.25:6789/0 -11377> 2013-04-03 10:42:13.806451 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 142) v1 -- ?+0 0x2e8b380 con 0x1bac160 -11376> 2013-04-03 10:42:13.806471 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos -11375> 2013-04-03 10:42:13.806486 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 -11374> 2013-04-03 10:42:13.806495 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -11373> 2013-04-03 10:42:13.806508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 preprocess_log log(1 entries) v1 from mon.0 -11372> 2013-04-03 10:42:13.806526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 nothing new -11371> 2013-04-03 10:42:13.806570 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos -11370> 2013-04-03 10:42:13.806585 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 -11369> 2013-04-03 10:42:13.806730 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11368> 2013-04-03 10:42:13.806752 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 142) v1 ==== 0+0+0 (0 0 0) 0x2e8b380 con 0x1bac160 -11367> 2013-04-03 10:42:13.806760 7fcdffcdb700 10 handle_log_ack log(last 142) v1 -11366> 2013-04-03 10:42:13.806761 7fcdffcdb700 10 logged 2013-04-03 10:42:12.744899 mon.0 10.252.24.25:6789/0 142 : [INF] pgmap v29076: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -11365> 2013-04-03 10:42:13.830483 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113716 ==== paxos(lease_ack lc 427437 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (109756242 0 0) 0x2df2280 con 0x208a840 -11364> 2013-04-03 10:42:13.830519 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos -11363> 2013-04-03 10:42:13.830536 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 -11362> 2013-04-03 10:42:13.830550 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11361> 2013-04-03 10:42:15.590870 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 132 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140417 v187) v2 ==== 289+0+0 (2609334374 0 0) 0x2ef9840 con 0x2118420 -11360> 2013-04-03 10:42:15.590948 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140417 v187) v2 from mds.? 10.252.58.31:6800/10908 -11359> 2013-04-03 10:42:15.590962 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140417 v187) v2 -- ?+0 0x2f56b80 con 0x2118420 -11358> 2013-04-03 10:42:16.806331 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427437 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2c6e500 -11357> 2013-04-03 10:42:16.807113 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113717 ==== paxos(lease_ack lc 427437 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (3404950387 0 0) 0x2c6e500 con 0x208a840 -11356> 2013-04-03 10:42:16.807188 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos -11355> 2013-04-03 10:42:16.807209 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 -11354> 2013-04-03 10:42:16.807226 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11353> 2013-04-03 10:42:16.937136 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113718 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 132 v187) v2) to leader v1 ==== 560+0+0 (3302759484 0 0) 0x20e3840 con 0x208a840 -11352> 2013-04-03 10:42:16.937156 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11351> 2013-04-03 10:42:16.937165 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e48c0 from 10.252.24.88:6789/0 -11350> 2013-04-03 10:42:16.937218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 132 v187) v2 from mds.0 10.252.24.25:6802/3144 -11349> 2013-04-03 10:42:16.937229 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 132 v187) v2 tid 17644) v2 -- ?+0 0x24f8d80 con 0x208a840 -11348> 2013-04-03 10:42:17.389458 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 132 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21788 v187) v2 ==== 289+0+0 (156575194 0 0) 0x2f568c0 con 0x200d420 -11347> 2013-04-03 10:42:17.389535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21788 v187) v2 from mds.? 10.252.24.88:6800/17213 -11346> 2013-04-03 10:42:17.389548 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21788 v187) v2 -- ?+0 0x20e3840 con 0x200d420 -11345> 2013-04-03 10:42:18.359832 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29077 check_down_pgs -11344> 2013-04-03 10:42:18.359866 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29077 v29077: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 12268B/s wr, 0op/s -11343> 2013-04-03 10:42:18.359894 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -11342> 2013-04-03 10:42:18.359918 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -11341> 2013-04-03 10:42:18.359954 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -11340> 2013-04-03 10:42:18.359985 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos -11339> 2013-04-03 10:42:18.360004 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 -11338> 2013-04-03 10:42:18.360013 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394074 log -11337> 2013-04-03 10:42:18.360030 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11336> 2013-04-03 10:42:18.360049 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -11335> 2013-04-03 10:42:19.590983 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 133 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140418 v187) v2 ==== 289+0+0 (4264856033 0 0) 0x2ef9580 con 0x2118420 -11334> 2013-04-03 10:42:19.591056 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140418 v187) v2 from mds.? 10.252.58.31:6800/10908 -11333> 2013-04-03 10:42:19.591069 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140418 v187) v2 -- ?+0 0x2f568c0 con 0x2118420 -11332> 2013-04-03 10:42:19.806397 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427437 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2c6e000 -11331> 2013-04-03 10:42:19.807179 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113719 ==== paxos(lease_ack lc 427437 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2483435111 0 0) 0x2c6e000 con 0x208a840 -11330> 2013-04-03 10:42:19.807234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos -11329> 2013-04-03 10:42:19.807254 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 -11328> 2013-04-03 10:42:19.807271 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11327> 2013-04-03 10:42:20.937262 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113720 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 133 v187) v2) to leader v1 ==== 560+0+0 (3449143871 0 0) 0x2ef9840 con 0x208a840 -11326> 2013-04-03 10:42:20.937284 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11325> 2013-04-03 10:42:20.937294 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4b80 from 10.252.24.88:6789/0 -11324> 2013-04-03 10:42:20.937355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 133 v187) v2 from mds.0 10.252.24.25:6802/3144 -11323> 2013-04-03 10:42:20.937370 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 133 v187) v2 tid 17645) v2 -- ?+0 0x24f8900 con 0x208a840 -11322> 2013-04-03 10:42:21.389570 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 133 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21789 v187) v2 ==== 289+0+0 (708549287 0 0) 0x2f56600 con 0x200d420 -11321> 2013-04-03 10:42:21.389651 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21789 v187) v2 from mds.? 10.252.24.88:6800/17213 -11320> 2013-04-03 10:42:21.389664 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21789 v187) v2 -- ?+0 0x2ef9840 con 0x200d420 -11319> 2013-04-03 10:42:22.806467 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427437 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2c6ea00 -11318> 2013-04-03 10:42:22.807262 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113721 ==== paxos(lease_ack lc 427437 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1849854991 0 0) 0x2c6ea00 con 0x208a840 -11317> 2013-04-03 10:42:22.807336 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos -11316> 2013-04-03 10:42:22.807356 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 -11315> 2013-04-03 10:42:22.807373 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11314> 2013-04-03 10:42:23.360196 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29077 v29077: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 12268B/s wr, 0op/s -11313> 2013-04-03 10:42:23.360237 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -11312> 2013-04-03 10:42:23.360260 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -11311> 2013-04-03 10:42:23.360305 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -11310> 2013-04-03 10:42:23.360335 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos -11309> 2013-04-03 10:42:23.360354 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 -11308> 2013-04-03 10:42:23.360363 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394074 log -11307> 2013-04-03 10:42:23.360514 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11306> 2013-04-03 10:42:23.360538 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -11305> 2013-04-03 10:42:23.591090 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 134 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140419 v187) v2 ==== 289+0+0 (3713676956 0 0) 0x2ef92c0 con 0x2118420 -11304> 2013-04-03 10:42:23.591142 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140419 v187) v2 from mds.? 10.252.58.31:6800/10908 -11303> 2013-04-03 10:42:23.591154 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140419 v187) v2 -- ?+0 0x2f56600 con 0x2118420 -11302> 2013-04-03 10:42:24.937492 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113722 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 134 v187) v2) to leader v1 ==== 560+0+0 (3595542586 0 0) 0x2ef9580 con 0x208a840 -11301> 2013-04-03 10:42:24.937514 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11300> 2013-04-03 10:42:24.937524 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4080 from 10.252.24.88:6789/0 -11299> 2013-04-03 10:42:24.937585 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 134 v187) v2 from mds.0 10.252.24.25:6802/3144 -11298> 2013-04-03 10:42:24.937597 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 134 v187) v2 tid 17646) v2 -- ?+0 0x24f8d80 con 0x208a840 -11297> 2013-04-03 10:42:25.389657 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 134 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21790 v187) v2 ==== 289+0+0 (1334336288 0 0) 0x2f56340 con 0x200d420 -11296> 2013-04-03 10:42:25.389736 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21790 v187) v2 from mds.? 10.252.24.88:6800/17213 -11295> 2013-04-03 10:42:25.389750 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21790 v187) v2 -- ?+0 0x2ef9580 con 0x200d420 -11294> 2013-04-03 10:42:25.806533 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427437 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29fda00 -11293> 2013-04-03 10:42:25.807273 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113723 ==== paxos(lease_ack lc 427437 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1989543577 0 0) 0x29fda00 con 0x208a840 -11292> 2013-04-03 10:42:25.807326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos -11291> 2013-04-03 10:42:25.807346 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 -11290> 2013-04-03 10:42:25.807363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11289> 2013-04-03 10:42:26.608933 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113724 ==== forward(pg_stats(1 pgs tid 43 v 605) v1) to leader v1 ==== 753+0+0 (748126512 0 0) 0x2ef92c0 con 0x208a840 -11288> 2013-04-03 10:42:26.608954 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -11287> 2013-04-03 10:42:26.608964 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -11286> 2013-04-03 10:42:26.609016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 preprocess_query pg_stats(1 pgs tid 43 v 605) v1 from osd.1 10.252.24.25:6805/3572 -11285> 2013-04-03 10:42:26.609041 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 prepare_update pg_stats(1 pgs tid 43 v 605) v1 from osd.1 10.252.24.25:6805/3572 -11284> 2013-04-03 10:42:26.609046 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 prepare_pg_stats pg_stats(1 pgs tid 43 v 605) v1 from osd.1 -11283> 2013-04-03 10:42:26.609052 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29077 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -11282> 2013-04-03 10:42:26.659236 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29077 encode_pending v 29078 -11281> 2013-04-03 10:42:26.659273 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427437) queue_proposal bl 650 bytes; ctx = 0x2c3d6c0 -11280> 2013-04-03 10:42:26.659282 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427437) propose_queued 427438 650 bytes -11279> 2013-04-03 10:42:26.660398 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427437 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd500 -11278> 2013-04-03 10:42:26.686857 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113725 ==== paxos(accept lc 427437 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1822683407 0 0) 0x29fd500 con 0x208a840 -11277> 2013-04-03 10:42:26.687211 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427438 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fda00 -11276> 2013-04-03 10:42:26.687241 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427438 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29fd000 -11275> 2013-04-03 10:42:26.687309 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29077 update_from_paxos applying incremental 29078 -11274> 2013-04-03 10:42:26.687324 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29078 v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s -11273> 2013-04-03 10:42:26.687344 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29078 send_pg_creates to 0 pgs -11272> 2013-04-03 10:42:26.687347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29078 update_logger -11271> 2013-04-03 10:42:26.687357 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29078 create_pending v 29079 -11270> 2013-04-03 10:42:26.687372 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29078 _updated_stats for osd.1 10.252.24.25:6805/3572 -11269> 2013-04-03 10:42:26.687379 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 43) v1 tid 17647) v2 -- ?+0 0x24f8900 con 0x208a840 -11268> 2013-04-03 10:42:26.687389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29078 check_osd_map already seen 605 >= 605 -11267> 2013-04-03 10:42:26.687392 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29078 update_logger -11266> 2013-04-03 10:42:26.687402 7fcdffcdb700 0 log [INF] : pgmap v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s -11265> 2013-04-03 10:42:26.687403 7fcdffcdb700 10 send_log to self -11264> 2013-04-03 10:42:26.687404 7fcdffcdb700 10 log_queue is 2 last_log 144 sent 142 num 2 unsent 2 sending 2 -11263> 2013-04-03 10:42:26.687406 7fcdffcdb700 10 will send 2013-04-03 10:42:13.774896 mon.0 10.252.24.25:6789/0 143 : [INF] pgmap v29077: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 12268B/s wr, 0op/s -11262> 2013-04-03 10:42:26.687412 7fcdffcdb700 10 will send 2013-04-03 10:42:26.687403 mon.0 10.252.24.25:6789/0 144 : [INF] pgmap v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s -11261> 2013-04-03 10:42:26.687417 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b73600 -11260> 2013-04-03 10:42:26.687466 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos -11259> 2013-04-03 10:42:26.687485 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 -11258> 2013-04-03 10:42:26.687504 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11257> 2013-04-03 10:42:26.687524 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73600 con 0x1bac160 -11256> 2013-04-03 10:42:26.687546 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos -11255> 2013-04-03 10:42:26.687560 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 -11254> 2013-04-03 10:42:26.687569 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -11253> 2013-04-03 10:42:26.687585 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 preprocess_log log(2 entries) v1 from mon.0 -11252> 2013-04-03 10:42:26.687617 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -11251> 2013-04-03 10:42:26.687632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 prepare_log log(2 entries) v1 from mon.0 -11250> 2013-04-03 10:42:26.687643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 logging 2013-04-03 10:42:13.774896 mon.0 10.252.24.25:6789/0 143 : [INF] pgmap v29077: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 12268B/s wr, 0op/s -11249> 2013-04-03 10:42:26.687664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 logging 2013-04-03 10:42:26.687403 mon.0 10.252.24.25:6789/0 144 : [INF] pgmap v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s -11248> 2013-04-03 10:42:26.698885 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113726 ==== paxos(lease_ack lc 427438 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1821082874 0 0) 0x29fd000 con 0x208a840 -11247> 2013-04-03 10:42:26.698937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos -11246> 2013-04-03 10:42:26.698955 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394074 update_from_paxos version 394074 summary v 394074 -11245> 2013-04-03 10:42:26.698971 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11244> 2013-04-03 10:42:27.591222 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 135 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140420 v187) v2 ==== 289+0+0 (894634735 0 0) 0x2ef9000 con 0x2118420 -11243> 2013-04-03 10:42:27.591297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140420 v187) v2 from mds.? 10.252.58.31:6800/10908 -11242> 2013-04-03 10:42:27.591309 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140420 v187) v2 -- ?+0 0x2ef92c0 con 0x2118420 -11241> 2013-04-03 10:42:27.687042 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394074 encode_pending v394075 -11240> 2013-04-03 10:42:27.687068 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427438) queue_proposal bl 727 bytes; ctx = 0x2c3d6e0 -11239> 2013-04-03 10:42:27.687076 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427382..427438) propose_queued 427439 727 bytes -11238> 2013-04-03 10:42:27.688232 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427438 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fdc80 -11237> 2013-04-03 10:42:27.739873 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113727 ==== paxos(accept lc 427438 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2527038860 0 0) 0x29fdc80 con 0x208a840 -11236> 2013-04-03 10:42:27.740200 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427439 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29fd000 -11235> 2013-04-03 10:42:27.740227 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427439 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29fd500 -11234> 2013-04-03 10:42:27.740266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos -11233> 2013-04-03 10:42:27.740293 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos version 394075 summary v 394074 -11232> 2013-04-03 10:42:27.740307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos latest full 393798 -11231> 2013-04-03 10:42:27.740326 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394075 update_from_paxos applying incremental log 394075 2013-04-03 10:42:13.774896 mon.0 10.252.24.25:6789/0 143 : [INF] pgmap v29077: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 12268B/s wr, 0op/s -11230> 2013-04-03 10:42:27.740356 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394075 update_from_paxos applying incremental log 394075 2013-04-03 10:42:26.687403 mon.0 10.252.24.25:6789/0 144 : [INF] pgmap v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s -11229> 2013-04-03 10:42:27.740405 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 check_subs -11228> 2013-04-03 10:42:27.740439 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 create_pending v 394076 -11227> 2013-04-03 10:42:27.740461 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394075 _updated_log for mon.0 10.252.24.25:6789/0 -11226> 2013-04-03 10:42:27.740473 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 144) v1 -- ?+0 0x2224380 con 0x1bac160 -11225> 2013-04-03 10:42:27.740524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos -11224> 2013-04-03 10:42:27.740539 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos version 394075 summary v 394075 -11223> 2013-04-03 10:42:27.740558 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11222> 2013-04-03 10:42:27.740578 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 144) v1 ==== 0+0+0 (0 0 0) 0x2224380 con 0x1bac160 -11221> 2013-04-03 10:42:27.740586 7fcdffcdb700 10 handle_log_ack log(last 144) v1 -11220> 2013-04-03 10:42:27.740588 7fcdffcdb700 10 logged 2013-04-03 10:42:13.774896 mon.0 10.252.24.25:6789/0 143 : [INF] pgmap v29077: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 12268B/s wr, 0op/s -11219> 2013-04-03 10:42:27.740594 7fcdffcdb700 10 logged 2013-04-03 10:42:26.687403 mon.0 10.252.24.25:6789/0 144 : [INF] pgmap v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s -11218> 2013-04-03 10:42:27.765194 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113728 ==== paxos(lease_ack lc 427439 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1803185555 0 0) 0x2c6ec80 con 0x208a840 -11217> 2013-04-03 10:42:27.765232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos -11216> 2013-04-03 10:42:27.765249 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos version 394075 summary v 394075 -11215> 2013-04-03 10:42:27.765266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11214> 2013-04-03 10:42:28.360688 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29078 check_down_pgs -11213> 2013-04-03 10:42:28.360718 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29078 v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s -11212> 2013-04-03 10:42:28.360746 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -11211> 2013-04-03 10:42:28.360770 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -11210> 2013-04-03 10:42:28.360806 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -11209> 2013-04-03 10:42:28.360837 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos -11208> 2013-04-03 10:42:28.360856 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394075 update_from_paxos version 394075 summary v 394075 -11207> 2013-04-03 10:42:28.360865 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394075 log -11206> 2013-04-03 10:42:28.360882 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -11205> 2013-04-03 10:42:28.360905 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -11204> 2013-04-03 10:42:28.597699 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427382..427439) queue_proposal bl 1467 bytes; ctx = 0x2c3d6d0 -11203> 2013-04-03 10:42:28.694875 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113729 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (424405211 0 0) 0x2ce28c0 con 0x208a840 -11202> 2013-04-03 10:42:28.694893 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -11201> 2013-04-03 10:42:28.694899 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -11200> 2013-04-03 10:42:28.694910 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 -11199> 2013-04-03 10:42:28.694914 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( start_reply flags( reply-to(mon.2 10.252.24.88:6789/0) ) ) v1 -- ?+0 0x2ef9000 -11198> 2013-04-03 10:42:28.937468 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113730 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 135 v187) v2) to leader v1 ==== 560+0+0 (2198346688 0 0) 0x2ce2b80 con 0x208a840 -11197> 2013-04-03 10:42:28.937483 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11196> 2013-04-03 10:42:28.937493 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ce2340 from 10.252.24.88:6789/0 -11195> 2013-04-03 10:42:28.937536 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 135 v187) v2 from mds.0 10.252.24.25:6802/3144 -11194> 2013-04-03 10:42:28.938054 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 135 v187) v2 tid 17648) v2 -- ?+0 0x24f8240 con 0x208a840 -11193> 2013-04-03 10:42:29.389812 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 135 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21791 v187) v2 ==== 289+0+0 (1827071069 0 0) 0x2f56080 con 0x200d420 -11192> 2013-04-03 10:42:29.389894 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21791 v187) v2 from mds.? 10.252.24.88:6800/17213 -11191> 2013-04-03 10:42:29.389909 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21791 v187) v2 -- ?+0 0x2ce2b80 con 0x200d420 -11190> 2013-04-03 10:42:30.740303 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427439 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29b4280 -11189> 2013-04-03 10:42:30.741064 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113731 ==== paxos(lease_ack lc 427439 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2921536468 0 0) 0x29b4280 con 0x208a840 -11188> 2013-04-03 10:42:30.741134 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos -11187> 2013-04-03 10:42:30.741155 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 -11186> 2013-04-03 10:42:30.741173 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11185> 2013-04-03 10:42:31.591352 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 136 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140421 v187) v2 ==== 289+0+0 (373142930 0 0) 0x2f7ab80 con 0x2118420 -11184> 2013-04-03 10:42:31.591429 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140421 v187) v2 from mds.? 10.252.58.31:6800/10908 -11183> 2013-04-03 10:42:31.591444 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140421 v187) v2 -- ?+0 0x2f56080 con 0x2118420 -11182> 2013-04-03 10:42:32.937753 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113732 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 136 v187) v2) to leader v1 ==== 560+0+0 (827627006 0 0) 0x2ce28c0 con 0x208a840 -11181> 2013-04-03 10:42:32.937773 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11180> 2013-04-03 10:42:32.937784 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ce1840 from 10.252.24.88:6789/0 -11179> 2013-04-03 10:42:32.937839 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 136 v187) v2 from mds.0 10.252.24.25:6802/3144 -11178> 2013-04-03 10:42:32.937854 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 136 v187) v2 tid 17649) v2 -- ?+0 0x24f8900 con 0x208a840 -11177> 2013-04-03 10:42:33.361055 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s -11176> 2013-04-03 10:42:33.361097 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -11175> 2013-04-03 10:42:33.361121 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -11174> 2013-04-03 10:42:33.361173 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -11173> 2013-04-03 10:42:33.361206 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos -11172> 2013-04-03 10:42:33.361225 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 -11171> 2013-04-03 10:42:33.361235 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 log -11170> 2013-04-03 10:42:33.361252 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11169> 2013-04-03 10:42:33.361271 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -11168> 2013-04-03 10:42:33.389902 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 136 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21792 v187) v2 ==== 289+0+0 (3710090081 0 0) 0x2f55dc0 con 0x200d420 -11167> 2013-04-03 10:42:33.389948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21792 v187) v2 from mds.? 10.252.24.88:6800/17213 -11166> 2013-04-03 10:42:33.389960 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21792 v187) v2 -- ?+0 0x2ce28c0 con 0x200d420 -11165> 2013-04-03 10:42:33.740369 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427439 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29b4780 -11164> 2013-04-03 10:42:33.741188 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113733 ==== paxos(lease_ack lc 427439 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (2126396973 0 0) 0x29b4780 con 0x208a840 -11163> 2013-04-03 10:42:33.741231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos -11162> 2013-04-03 10:42:33.741249 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 -11161> 2013-04-03 10:42:33.741264 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11160> 2013-04-03 10:42:35.591447 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 137 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140422 v187) v2 ==== 289+0+0 (1938715669 0 0) 0x2f7a8c0 con 0x2118420 -11159> 2013-04-03 10:42:35.591525 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140422 v187) v2 from mds.? 10.252.58.31:6800/10908 -11158> 2013-04-03 10:42:35.591540 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140422 v187) v2 -- ?+0 0x2f55dc0 con 0x2118420 -11157> 2013-04-03 10:42:36.740440 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427439 fc 427382 pn 0 opn 0) v3 -- ?+0 0x29b4c80 -11156> 2013-04-03 10:42:36.741256 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113734 ==== paxos(lease_ack lc 427439 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (938474504 0 0) 0x29b4c80 con 0x208a840 -11155> 2013-04-03 10:42:36.741340 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos -11154> 2013-04-03 10:42:36.741361 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 -11153> 2013-04-03 10:42:36.741380 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11152> 2013-04-03 10:42:36.937769 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113735 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 137 v187) v2) to leader v1 ==== 560+0+0 (2687883673 0 0) 0x2f7ab80 con 0x208a840 -11151> 2013-04-03 10:42:36.937782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11150> 2013-04-03 10:42:36.937792 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ce1000 from 10.252.24.88:6789/0 -11149> 2013-04-03 10:42:36.937824 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 137 v187) v2 from mds.0 10.252.24.25:6802/3144 -11148> 2013-04-03 10:42:36.937839 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 137 v187) v2 tid 17650) v2 -- ?+0 0x24f8240 con 0x208a840 -11147> 2013-04-03 10:42:37.390009 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 137 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21793 v187) v2 ==== 289+0+0 (4266509340 0 0) 0x2f55b00 con 0x200d420 -11146> 2013-04-03 10:42:37.390092 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21793 v187) v2 from mds.? 10.252.24.88:6800/17213 -11145> 2013-04-03 10:42:37.390107 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21793 v187) v2 -- ?+0 0x2f7ab80 con 0x200d420 -11144> 2013-04-03 10:42:38.361420 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 v29078: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2052B/s wr, 0op/s -11143> 2013-04-03 10:42:38.361461 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -11142> 2013-04-03 10:42:38.361485 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -11141> 2013-04-03 10:42:38.361531 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -11140> 2013-04-03 10:42:38.361563 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos -11139> 2013-04-03 10:42:38.361587 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 -11138> 2013-04-03 10:42:38.361597 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 log -11137> 2013-04-03 10:42:38.361614 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11136> 2013-04-03 10:42:38.361633 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -11135> 2013-04-03 10:42:39.591561 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 138 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140423 v187) v2 ==== 289+0+0 (1356909416 0 0) 0x2f7a600 con 0x2118420 -11134> 2013-04-03 10:42:39.591769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140423 v187) v2 from mds.? 10.252.58.31:6800/10908 -11133> 2013-04-03 10:42:39.591787 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140423 v187) v2 -- ?+0 0x2f55b00 con 0x2118420 -11132> 2013-04-03 10:42:39.740514 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427439 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2116a00 -11131> 2013-04-03 10:42:39.741300 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113736 ==== paxos(lease_ack lc 427439 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (1066844284 0 0) 0x2116a00 con 0x208a840 -11130> 2013-04-03 10:42:39.741352 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos -11129> 2013-04-03 10:42:39.741372 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 -11128> 2013-04-03 10:42:39.741390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11127> 2013-04-03 10:42:40.937913 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113737 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 138 v187) v2) to leader v1 ==== 560+0+0 (600254968 0 0) 0x2f7a8c0 con 0x208a840 -11126> 2013-04-03 10:42:40.937934 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11125> 2013-04-03 10:42:40.937945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273b600 from 10.252.24.88:6789/0 -11124> 2013-04-03 10:42:40.937999 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 138 v187) v2 from mds.0 10.252.24.25:6802/3144 -11123> 2013-04-03 10:42:40.938012 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 138 v187) v2 tid 17651) v2 -- ?+0 0x24f8900 con 0x208a840 -11122> 2013-04-03 10:42:41.390152 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 138 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21794 v187) v2 ==== 289+0+0 (2617116059 0 0) 0x2f55840 con 0x200d420 -11121> 2013-04-03 10:42:41.390229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21794 v187) v2 from mds.? 10.252.24.88:6800/17213 -11120> 2013-04-03 10:42:41.390244 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21794 v187) v2 -- ?+0 0x2f7a8c0 con 0x200d420 -11119> 2013-04-03 10:42:42.740585 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427439 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2116500 -11118> 2013-04-03 10:42:42.741375 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113738 ==== paxos(lease_ack lc 427439 fc 427382 pn 0 opn 0) v3 ==== 80+0+0 (131015549 0 0) 0x2116500 con 0x208a840 -11117> 2013-04-03 10:42:42.741454 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos -11116> 2013-04-03 10:42:42.741475 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 -11115> 2013-04-03 10:42:42.741493 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11114> 2013-04-03 10:42:43.254586 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113739 ==== forward(pg_stats(1 pgs tid 24 v 605) v1) to leader v1 ==== 757+0+0 (1205735369 0 0) 0x2f7a600 con 0x208a840 -11113> 2013-04-03 10:42:43.254614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -11112> 2013-04-03 10:42:43.254625 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -11111> 2013-04-03 10:42:43.254676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 preprocess_query pg_stats(1 pgs tid 24 v 605) v1 from osd.2 10.252.24.30:6801/5656 -11110> 2013-04-03 10:42:43.254701 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 prepare_update pg_stats(1 pgs tid 24 v 605) v1 from osd.2 10.252.24.30:6801/5656 -11109> 2013-04-03 10:42:43.254706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 prepare_pg_stats pg_stats(1 pgs tid 24 v 605) v1 from osd.2 -11108> 2013-04-03 10:42:43.254712 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -11107> 2013-04-03 10:42:43.304895 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 encode_pending v 29079 -11106> 2013-04-03 10:42:43.304928 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427382..427439) queue_proposal bl 654 bytes; ctx = 0x2c3d940 -11105> 2013-04-03 10:42:43.304938 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427382..427439) propose_queued 427440 1467 bytes -11104> 2013-04-03 10:42:43.306455 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427439 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116780 -11103> 2013-04-03 10:42:43.350198 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113740 ==== paxos(accept lc 427439 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1594693696 0 0) 0x2116780 con 0x208a840 -11102> 2013-04-03 10:42:43.350544 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427440 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116500 -11101> 2013-04-03 10:42:43.350570 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427440 fc 427382 pn 0 opn 0) v3 -- ?+0 0x2116a00 -11100> 2013-04-03 10:42:43.350620 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427440) propose_queued 427441 654 bytes -11099> 2013-04-03 10:42:43.350828 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427440 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4c80 -11098> 2013-04-03 10:42:43.361753 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -11097> 2013-04-03 10:42:43.361790 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -11096> 2013-04-03 10:42:43.361814 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos -11095> 2013-04-03 10:42:43.361834 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 -11094> 2013-04-03 10:42:43.361843 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 log -11093> 2013-04-03 10:42:43.362491 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11092> 2013-04-03 10:42:43.362519 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -11091> 2013-04-03 10:42:43.375488 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113741 ==== paxos(lease_ack lc 427440 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3153519115 0 0) 0x29b4c80 con 0x208a840 -11090> 2013-04-03 10:42:43.387297 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113742 ==== paxos(accept lc 427440 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (137172833 0 0) 0x2116a00 con 0x208a840 -11089> 2013-04-03 10:42:43.387604 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427441 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4c80 -11088> 2013-04-03 10:42:43.387635 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427441 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2116780 -11087> 2013-04-03 10:42:43.387713 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29078 update_from_paxos applying incremental 29079 -11086> 2013-04-03 10:42:43.387734 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 v29079: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3186B/s wr, 0op/s -11085> 2013-04-03 10:42:43.387754 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 send_pg_creates to 0 pgs -11084> 2013-04-03 10:42:43.387759 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 update_logger -11083> 2013-04-03 10:42:43.387769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 create_pending v 29080 -11082> 2013-04-03 10:42:43.387779 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 _updated_stats for osd.2 10.252.24.30:6801/5656 -11081> 2013-04-03 10:42:43.387786 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 24) v1 tid 17652) v2 -- ?+0 0x24f8240 con 0x208a840 -11080> 2013-04-03 10:42:43.387800 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 check_osd_map already seen 605 >= 605 -11079> 2013-04-03 10:42:43.387804 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 update_logger -11078> 2013-04-03 10:42:43.387814 7fcdffcdb700 0 log [INF] : pgmap v29079: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3186B/s wr, 0op/s -11077> 2013-04-03 10:42:43.387816 7fcdffcdb700 10 send_log to self -11076> 2013-04-03 10:42:43.387817 7fcdffcdb700 10 log_queue is 1 last_log 145 sent 144 num 1 unsent 1 sending 1 -11075> 2013-04-03 10:42:43.387818 7fcdffcdb700 10 will send 2013-04-03 10:42:43.387816 mon.0 10.252.24.25:6789/0 145 : [INF] pgmap v29079: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3186B/s wr, 0op/s -11074> 2013-04-03 10:42:43.387825 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75200 -11073> 2013-04-03 10:42:43.387873 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos -11072> 2013-04-03 10:42:43.387891 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 -11071> 2013-04-03 10:42:43.387912 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11070> 2013-04-03 10:42:43.387933 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75200 con 0x1bac160 -11069> 2013-04-03 10:42:43.387958 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos -11068> 2013-04-03 10:42:43.387974 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 -11067> 2013-04-03 10:42:43.387983 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -11066> 2013-04-03 10:42:43.387995 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 preprocess_log log(1 entries) v1 from mon.0 -11065> 2013-04-03 10:42:43.388021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -11064> 2013-04-03 10:42:43.388034 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 prepare_log log(1 entries) v1 from mon.0 -11063> 2013-04-03 10:42:43.388044 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 logging 2013-04-03 10:42:43.387816 mon.0 10.252.24.25:6789/0 145 : [INF] pgmap v29079: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3186B/s wr, 0op/s -11062> 2013-04-03 10:42:43.399262 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113743 ==== paxos(lease_ack lc 427441 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (467076683 0 0) 0x2116780 con 0x208a840 -11061> 2013-04-03 10:42:43.399302 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos -11060> 2013-04-03 10:42:43.399319 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 update_from_paxos version 394075 summary v 394075 -11059> 2013-04-03 10:42:43.399336 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11058> 2013-04-03 10:42:43.591677 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 139 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140424 v187) v2 ==== 289+0+0 (2792389122 0 0) 0x2f7a340 con 0x2118420 -11057> 2013-04-03 10:42:43.591729 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140424 v187) v2 from mds.? 10.252.58.31:6800/10908 -11056> 2013-04-03 10:42:43.591740 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140424 v187) v2 -- ?+0 0x2f7a600 con 0x2118420 -11055> 2013-04-03 10:42:44.387499 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394075 encode_pending v394076 -11054> 2013-04-03 10:42:44.387530 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427441) queue_proposal bl 443 bytes; ctx = 0x2c3d960 -11053> 2013-04-03 10:42:44.387546 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427441) propose_queued 427442 443 bytes -11052> 2013-04-03 10:42:44.388555 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427441 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116280 -11051> 2013-04-03 10:42:44.415174 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113744 ==== paxos(accept lc 427441 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (257379118 0 0) 0x2116280 con 0x208a840 -11050> 2013-04-03 10:42:44.415520 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427442 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116780 -11049> 2013-04-03 10:42:44.415552 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427442 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2116a00 -11048> 2013-04-03 10:42:44.415589 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos -11047> 2013-04-03 10:42:44.415617 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394075 -11046> 2013-04-03 10:42:44.415637 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos latest full 393798 -11045> 2013-04-03 10:42:44.415656 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos applying incremental log 394076 2013-04-03 10:42:43.387816 mon.0 10.252.24.25:6789/0 145 : [INF] pgmap v29079: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3186B/s wr, 0op/s -11044> 2013-04-03 10:42:44.415706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 check_subs -11043> 2013-04-03 10:42:44.415739 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 create_pending v 394077 -11042> 2013-04-03 10:42:44.415762 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 _updated_log for mon.0 10.252.24.25:6789/0 -11041> 2013-04-03 10:42:44.415775 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 145) v1 -- ?+0 0x28a5540 con 0x1bac160 -11040> 2013-04-03 10:42:44.415835 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos -11039> 2013-04-03 10:42:44.415851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 -11038> 2013-04-03 10:42:44.415870 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11037> 2013-04-03 10:42:44.415894 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 145) v1 ==== 0+0+0 (0 0 0) 0x28a5540 con 0x1bac160 -11036> 2013-04-03 10:42:44.415902 7fcdffcdb700 10 handle_log_ack log(last 145) v1 -11035> 2013-04-03 10:42:44.415905 7fcdffcdb700 10 logged 2013-04-03 10:42:43.387816 mon.0 10.252.24.25:6789/0 145 : [INF] pgmap v29079: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3186B/s wr, 0op/s -11034> 2013-04-03 10:42:44.440402 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113745 ==== paxos(lease_ack lc 427442 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (637525179 0 0) 0x2116a00 con 0x208a840 -11033> 2013-04-03 10:42:44.440442 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos -11032> 2013-04-03 10:42:44.440459 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 -11031> 2013-04-03 10:42:44.440474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11030> 2013-04-03 10:42:44.938016 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113746 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 139 v187) v2) to leader v1 ==== 560+0+0 (3406790292 0 0) 0x273b080 con 0x208a840 -11029> 2013-04-03 10:42:44.938033 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11028> 2013-04-03 10:42:44.938043 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273adc0 from 10.252.24.88:6789/0 -11027> 2013-04-03 10:42:44.938079 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 139 v187) v2 from mds.0 10.252.24.25:6802/3144 -11026> 2013-04-03 10:42:44.938091 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 139 v187) v2 tid 17653) v2 -- ?+0 0x24f8900 con 0x208a840 -11025> 2013-04-03 10:42:45.390257 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 139 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21795 v187) v2 ==== 289+0+0 (3096492774 0 0) 0x2f55580 con 0x200d420 -11024> 2013-04-03 10:42:45.390336 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21795 v187) v2 from mds.? 10.252.24.88:6800/17213 -11023> 2013-04-03 10:42:45.390351 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21795 v187) v2 -- ?+0 0x273bb80 con 0x200d420 -11022> 2013-04-03 10:42:47.415631 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427442 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2116000 -11021> 2013-04-03 10:42:47.416511 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113747 ==== paxos(lease_ack lc 427442 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (1865780470 0 0) 0x2116000 con 0x208a840 -11020> 2013-04-03 10:42:47.416580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos -11019> 2013-04-03 10:42:47.416728 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 -11018> 2013-04-03 10:42:47.416748 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11017> 2013-04-03 10:42:47.591797 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 140 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140425 v187) v2 ==== 289+0+0 (2233386367 0 0) 0x2f7a080 con 0x2118420 -11016> 2013-04-03 10:42:47.591849 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140425 v187) v2 from mds.? 10.252.58.31:6800/10908 -11015> 2013-04-03 10:42:47.591861 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140425 v187) v2 -- ?+0 0x2f55580 con 0x2118420 -11014> 2013-04-03 10:42:48.362640 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 check_down_pgs -11013> 2013-04-03 10:42:48.362670 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 v29079: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3186B/s wr, 0op/s -11012> 2013-04-03 10:42:48.362698 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -11011> 2013-04-03 10:42:48.362724 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -11010> 2013-04-03 10:42:48.362761 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -11009> 2013-04-03 10:42:48.362793 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos -11008> 2013-04-03 10:42:48.362812 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 -11007> 2013-04-03 10:42:48.362822 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 log -11006> 2013-04-03 10:42:48.362839 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -11005> 2013-04-03 10:42:48.362859 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -11004> 2013-04-03 10:42:48.938145 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113748 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 140 v187) v2) to leader v1 ==== 560+0+0 (3226416804 0 0) 0x2f7a340 con 0x208a840 -11003> 2013-04-03 10:42:48.938163 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -11002> 2013-04-03 10:42:48.938174 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2996340 from 10.252.24.88:6789/0 -11001> 2013-04-03 10:42:48.938220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 140 v187) v2 from mds.0 10.252.24.25:6802/3144 -11000> 2013-04-03 10:42:48.938232 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 140 v187) v2 tid 17654) v2 -- ?+0 0x24f8240 con 0x208a840 -10999> 2013-04-03 10:42:49.390400 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 140 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21796 v187) v2 ==== 289+0+0 (1352238741 0 0) 0x2f552c0 con 0x200d420 -10998> 2013-04-03 10:42:49.390476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21796 v187) v2 from mds.? 10.252.24.88:6800/17213 -10997> 2013-04-03 10:42:49.390494 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21796 v187) v2 -- ?+0 0x2f7a340 con 0x200d420 -10996> 2013-04-03 10:42:50.415701 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427442 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2df2500 -10995> 2013-04-03 10:42:50.416595 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113749 ==== paxos(lease_ack lc 427442 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3137950211 0 0) 0x2df2500 con 0x208a840 -10994> 2013-04-03 10:42:50.416669 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos -10993> 2013-04-03 10:42:50.416896 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 -10992> 2013-04-03 10:42:50.416916 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10991> 2013-04-03 10:42:51.591933 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 141 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140426 v187) v2 ==== 289+0+0 (3769435384 0 0) 0x2f79dc0 con 0x2118420 -10990> 2013-04-03 10:42:51.592008 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140426 v187) v2 from mds.? 10.252.58.31:6800/10908 -10989> 2013-04-03 10:42:51.592022 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140426 v187) v2 -- ?+0 0x2f552c0 con 0x2118420 -10988> 2013-04-03 10:42:51.611582 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113750 ==== forward(pg_stats(1 pgs tid 44 v 605) v1) to leader v1 ==== 753+0+0 (284102191 0 0) 0x2f7a080 con 0x208a840 -10987> 2013-04-03 10:42:51.611594 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -10986> 2013-04-03 10:42:51.611603 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -10985> 2013-04-03 10:42:51.611637 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 preprocess_query pg_stats(1 pgs tid 44 v 605) v1 from osd.1 10.252.24.25:6805/3572 -10984> 2013-04-03 10:42:51.611662 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 prepare_update pg_stats(1 pgs tid 44 v 605) v1 from osd.1 10.252.24.25:6805/3572 -10983> 2013-04-03 10:42:51.611668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 prepare_pg_stats pg_stats(1 pgs tid 44 v 605) v1 from osd.1 -10982> 2013-04-03 10:42:51.611675 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -10981> 2013-04-03 10:42:51.661858 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 encode_pending v 29080 -10980> 2013-04-03 10:42:51.661895 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427442) queue_proposal bl 650 bytes; ctx = 0x2c3d920 -10979> 2013-04-03 10:42:51.661904 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427442) propose_queued 427443 650 bytes -10978> 2013-04-03 10:42:51.663199 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427442 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2c80 -10977> 2013-04-03 10:42:51.691547 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113751 ==== paxos(accept lc 427442 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1578548928 0 0) 0x2df2c80 con 0x208a840 -10976> 2013-04-03 10:42:51.691929 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427443 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2500 -10975> 2013-04-03 10:42:51.691958 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427443 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2116000 -10974> 2013-04-03 10:42:51.692024 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29079 update_from_paxos applying incremental 29080 -10973> 2013-04-03 10:42:51.692044 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 v29080: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3767B/s wr, 0op/s -10972> 2013-04-03 10:42:51.692066 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 send_pg_creates to 0 pgs -10971> 2013-04-03 10:42:51.692070 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 update_logger -10970> 2013-04-03 10:42:51.692079 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 create_pending v 29081 -10969> 2013-04-03 10:42:51.692089 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 _updated_stats for osd.1 10.252.24.25:6805/3572 -10968> 2013-04-03 10:42:51.692096 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 44) v1 tid 17655) v2 -- ?+0 0x24f8900 con 0x208a840 -10967> 2013-04-03 10:42:51.692105 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 check_osd_map already seen 605 >= 605 -10966> 2013-04-03 10:42:51.692109 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 update_logger -10965> 2013-04-03 10:42:51.692120 7fcdffcdb700 0 log [INF] : pgmap v29080: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3767B/s wr, 0op/s -10964> 2013-04-03 10:42:51.692122 7fcdffcdb700 10 send_log to self -10963> 2013-04-03 10:42:51.692122 7fcdffcdb700 10 log_queue is 1 last_log 146 sent 145 num 1 unsent 1 sending 1 -10962> 2013-04-03 10:42:51.692124 7fcdffcdb700 10 will send 2013-04-03 10:42:51.692121 mon.0 10.252.24.25:6789/0 146 : [INF] pgmap v29080: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3767B/s wr, 0op/s -10961> 2013-04-03 10:42:51.692130 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287c000 -10960> 2013-04-03 10:42:51.692175 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos -10959> 2013-04-03 10:42:51.692195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 -10958> 2013-04-03 10:42:51.692215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10957> 2013-04-03 10:42:51.692236 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287c000 con 0x1bac160 -10956> 2013-04-03 10:42:51.692258 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos -10955> 2013-04-03 10:42:51.692274 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 -10954> 2013-04-03 10:42:51.692286 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10953> 2013-04-03 10:42:51.692297 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 preprocess_log log(1 entries) v1 from mon.0 -10952> 2013-04-03 10:42:51.692322 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10951> 2013-04-03 10:42:51.692335 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 prepare_log log(1 entries) v1 from mon.0 -10950> 2013-04-03 10:42:51.692345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 logging 2013-04-03 10:42:51.692121 mon.0 10.252.24.25:6789/0 146 : [INF] pgmap v29080: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3767B/s wr, 0op/s -10949> 2013-04-03 10:42:51.704786 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113752 ==== paxos(lease_ack lc 427443 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3625571684 0 0) 0x2116000 con 0x208a840 -10948> 2013-04-03 10:42:51.704825 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos -10947> 2013-04-03 10:42:51.704843 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 update_from_paxos version 394076 summary v 394076 -10946> 2013-04-03 10:42:51.704858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10945> 2013-04-03 10:42:51.776940 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113753 ==== forward(pg_stats(1 pgs tid 30454 v 605) v1) to leader v1 ==== 757+0+0 (4025731707 0 0) 0x2995580 con 0x208a840 -10944> 2013-04-03 10:42:51.776952 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -10943> 2013-04-03 10:42:51.776959 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -10942> 2013-04-03 10:42:51.776987 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 preprocess_query pg_stats(1 pgs tid 30454 v 605) v1 from osd.3 10.252.24.88:6801/17538 -10941> 2013-04-03 10:42:51.777009 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 prepare_update pg_stats(1 pgs tid 30454 v 605) v1 from osd.3 10.252.24.88:6801/17538 -10940> 2013-04-03 10:42:51.777014 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 prepare_pg_stats pg_stats(1 pgs tid 30454 v 605) v1 from osd.3 -10939> 2013-04-03 10:42:51.777019 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -10938> 2013-04-03 10:42:52.691742 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 encode_pending v 29081 -10937> 2013-04-03 10:42:52.691788 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427443) queue_proposal bl 654 bytes; ctx = 0x2c3d830 -10936> 2013-04-03 10:42:52.691798 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427443) propose_queued 427444 654 bytes -10935> 2013-04-03 10:42:52.692872 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427443 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2a00 -10934> 2013-04-03 10:42:52.692981 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394076 encode_pending v394077 -10933> 2013-04-03 10:42:52.693008 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427429..427443) queue_proposal bl 443 bytes; ctx = 0x2c3d950 -10932> 2013-04-03 10:42:52.693016 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427429..427443) propose_new_value not active; proposal queued -10931> 2013-04-03 10:42:52.720878 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113754 ==== paxos(accept lc 427443 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (900622250 0 0) 0x2df2a00 con 0x208a840 -10930> 2013-04-03 10:42:52.721163 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427444 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116000 -10929> 2013-04-03 10:42:52.721197 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427444 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2df2c80 -10928> 2013-04-03 10:42:52.721273 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29080 update_from_paxos applying incremental 29081 -10927> 2013-04-03 10:42:52.721293 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 v29081: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -10926> 2013-04-03 10:42:52.721313 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 send_pg_creates to 0 pgs -10925> 2013-04-03 10:42:52.721317 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 update_logger -10924> 2013-04-03 10:42:52.721326 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 create_pending v 29082 -10923> 2013-04-03 10:42:52.721336 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 _updated_stats for osd.3 10.252.24.88:6801/17538 -10922> 2013-04-03 10:42:52.721343 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30454) v1 tid 17656) v2 -- ?+0 0x24f8240 con 0x208a840 -10921> 2013-04-03 10:42:52.721354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 check_osd_map already seen 605 >= 605 -10920> 2013-04-03 10:42:52.721357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 update_logger -10919> 2013-04-03 10:42:52.721368 7fcdffcdb700 0 log [INF] : pgmap v29081: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -10918> 2013-04-03 10:42:52.721369 7fcdffcdb700 10 send_log to self -10917> 2013-04-03 10:42:52.721370 7fcdffcdb700 10 log_queue is 2 last_log 147 sent 146 num 2 unsent 1 sending 1 -10916> 2013-04-03 10:42:52.721372 7fcdffcdb700 10 will send 2013-04-03 10:42:51.692121 mon.0 10.252.24.25:6789/0 146 : [INF] pgmap v29080: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3767B/s wr, 0op/s -10915> 2013-04-03 10:42:52.721378 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73e00 -10914> 2013-04-03 10:42:52.721404 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427444) propose_queued 427445 443 bytes -10913> 2013-04-03 10:42:52.721713 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427444 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116a00 -10912> 2013-04-03 10:42:52.721745 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73e00 con 0x1bac160 -10911> 2013-04-03 10:42:52.744949 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113755 ==== paxos(lease_ack lc 427444 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3237374613 0 0) 0x2116a00 con 0x208a840 -10910> 2013-04-03 10:42:52.757900 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113756 ==== paxos(accept lc 427444 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2367880077 0 0) 0x2df2c80 con 0x208a840 -10909> 2013-04-03 10:42:52.758193 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427445 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116a00 -10908> 2013-04-03 10:42:52.758226 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427445 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2df2a00 -10907> 2013-04-03 10:42:52.758267 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos -10906> 2013-04-03 10:42:52.758292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394076 -10905> 2013-04-03 10:42:52.758309 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos latest full 393798 -10904> 2013-04-03 10:42:52.758329 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos applying incremental log 394077 2013-04-03 10:42:51.692121 mon.0 10.252.24.25:6789/0 146 : [INF] pgmap v29080: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3767B/s wr, 0op/s -10903> 2013-04-03 10:42:52.758374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 check_subs -10902> 2013-04-03 10:42:52.758406 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 create_pending v 394078 -10901> 2013-04-03 10:42:52.758429 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 _updated_log for mon.0 10.252.24.25:6789/0 -10900> 2013-04-03 10:42:52.758442 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 146) v1 -- ?+0 0x28a58c0 con 0x1bac160 -10899> 2013-04-03 10:42:52.758462 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos -10898> 2013-04-03 10:42:52.758477 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 -10897> 2013-04-03 10:42:52.758487 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10896> 2013-04-03 10:42:52.758499 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 preprocess_log log(1 entries) v1 from mon.0 -10895> 2013-04-03 10:42:52.758519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 nothing new -10894> 2013-04-03 10:42:52.758568 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos -10893> 2013-04-03 10:42:52.758584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 -10892> 2013-04-03 10:42:52.758603 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10891> 2013-04-03 10:42:52.758624 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 146) v1 ==== 0+0+0 (0 0 0) 0x28a58c0 con 0x1bac160 -10890> 2013-04-03 10:42:52.758634 7fcdffcdb700 10 handle_log_ack log(last 146) v1 -10889> 2013-04-03 10:42:52.758637 7fcdffcdb700 10 logged 2013-04-03 10:42:51.692121 mon.0 10.252.24.25:6789/0 146 : [INF] pgmap v29080: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3767B/s wr, 0op/s -10888> 2013-04-03 10:42:52.770031 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113757 ==== paxos(lease_ack lc 427445 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3931503425 0 0) 0x2df2a00 con 0x208a840 -10887> 2013-04-03 10:42:52.770066 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos -10886> 2013-04-03 10:42:52.770083 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 -10885> 2013-04-03 10:42:52.770099 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10884> 2013-04-03 10:42:52.938329 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113758 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 141 v187) v2) to leader v1 ==== 560+0+0 (266964104 0 0) 0x2cc42c0 con 0x208a840 -10883> 2013-04-03 10:42:52.938352 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -10882> 2013-04-03 10:42:52.938363 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc4dc0 from 10.252.24.88:6789/0 -10881> 2013-04-03 10:42:52.938417 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 141 v187) v2 from mds.0 10.252.24.25:6802/3144 -10880> 2013-04-03 10:42:52.938429 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 141 v187) v2 tid 17657) v2 -- ?+0 0x24f8900 con 0x208a840 -10879> 2013-04-03 10:42:53.362979 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 check_down_pgs -10878> 2013-04-03 10:42:53.363010 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 v29081: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -10877> 2013-04-03 10:42:53.363040 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -10876> 2013-04-03 10:42:53.363066 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -10875> 2013-04-03 10:42:53.363104 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -10874> 2013-04-03 10:42:53.363136 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos -10873> 2013-04-03 10:42:53.363157 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 -10872> 2013-04-03 10:42:53.363167 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 log -10871> 2013-04-03 10:42:53.363185 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10870> 2013-04-03 10:42:53.363555 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -10869> 2013-04-03 10:42:53.390488 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 141 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21797 v187) v2 ==== 289+0+0 (1945582056 0 0) 0x2f55000 con 0x200d420 -10868> 2013-04-03 10:42:53.390555 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21797 v187) v2 from mds.? 10.252.24.88:6800/17213 -10867> 2013-04-03 10:42:53.390569 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21797 v187) v2 -- ?+0 0x2cc42c0 con 0x200d420 -10866> 2013-04-03 10:42:55.592052 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 142 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140427 v187) v2 ==== 289+0+0 (3284384645 0 0) 0x2f79b00 con 0x2118420 -10865> 2013-04-03 10:42:55.592131 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140427 v187) v2 from mds.? 10.252.58.31:6800/10908 -10864> 2013-04-03 10:42:55.592145 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140427 v187) v2 -- ?+0 0x2f55000 con 0x2118420 -10863> 2013-04-03 10:42:55.758297 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427445 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2df2280 -10862> 2013-04-03 10:42:55.759069 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113759 ==== paxos(lease_ack lc 427445 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (4163132943 0 0) 0x2df2280 con 0x208a840 -10861> 2013-04-03 10:42:55.759124 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos -10860> 2013-04-03 10:42:55.759148 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 -10859> 2013-04-03 10:42:55.759166 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10858> 2013-04-03 10:42:56.938365 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113760 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 142 v187) v2) to leader v1 ==== 560+0+0 (338947725 0 0) 0x2995580 con 0x208a840 -10857> 2013-04-03 10:42:56.938386 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -10856> 2013-04-03 10:42:56.938397 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc5340 from 10.252.24.88:6789/0 -10855> 2013-04-03 10:42:56.938452 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 142 v187) v2 from mds.0 10.252.24.25:6802/3144 -10854> 2013-04-03 10:42:56.938464 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 142 v187) v2 tid 17658) v2 -- ?+0 0x24f8240 con 0x208a840 -10853> 2013-04-03 10:42:57.390643 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 142 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21798 v187) v2 ==== 289+0+0 (373618799 0 0) 0x2ef92c0 con 0x200d420 -10852> 2013-04-03 10:42:57.390716 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21798 v187) v2 from mds.? 10.252.24.88:6800/17213 -10851> 2013-04-03 10:42:57.390730 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21798 v187) v2 -- ?+0 0x2995580 con 0x200d420 -10850> 2013-04-03 10:42:58.256562 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113761 ==== forward(pg_stats(1 pgs tid 25 v 605) v1) to leader v1 ==== 757+0+0 (970802496 0 0) 0x2f79b00 con 0x208a840 -10849> 2013-04-03 10:42:58.256579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -10848> 2013-04-03 10:42:58.256588 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -10847> 2013-04-03 10:42:58.256626 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 preprocess_query pg_stats(1 pgs tid 25 v 605) v1 from osd.2 10.252.24.30:6801/5656 -10846> 2013-04-03 10:42:58.256649 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 prepare_update pg_stats(1 pgs tid 25 v 605) v1 from osd.2 10.252.24.30:6801/5656 -10845> 2013-04-03 10:42:58.256654 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 prepare_pg_stats pg_stats(1 pgs tid 25 v 605) v1 from osd.2 -10844> 2013-04-03 10:42:58.256660 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -10843> 2013-04-03 10:42:58.306849 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 encode_pending v 29082 -10842> 2013-04-03 10:42:58.306886 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427445) queue_proposal bl 654 bytes; ctx = 0x1b91f50 -10841> 2013-04-03 10:42:58.306896 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427445) propose_queued 427446 654 bytes -10840> 2013-04-03 10:42:58.307936 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427445 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2780 -10839> 2013-04-03 10:42:58.332295 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113762 ==== paxos(accept lc 427445 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2588887629 0 0) 0x2df2780 con 0x208a840 -10838> 2013-04-03 10:42:58.332630 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427446 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2280 -10837> 2013-04-03 10:42:58.332663 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427446 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2df2a00 -10836> 2013-04-03 10:42:58.332737 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29081 update_from_paxos applying incremental 29082 -10835> 2013-04-03 10:42:58.332754 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 v29082: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3236B/s wr, 0op/s -10834> 2013-04-03 10:42:58.332774 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 send_pg_creates to 0 pgs -10833> 2013-04-03 10:42:58.332779 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 update_logger -10832> 2013-04-03 10:42:58.332790 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 create_pending v 29083 -10831> 2013-04-03 10:42:58.332801 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 _updated_stats for osd.2 10.252.24.30:6801/5656 -10830> 2013-04-03 10:42:58.332809 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 25) v1 tid 17659) v2 -- ?+0 0x24f8900 con 0x208a840 -10829> 2013-04-03 10:42:58.332822 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 check_osd_map already seen 605 >= 605 -10828> 2013-04-03 10:42:58.332826 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 update_logger -10827> 2013-04-03 10:42:58.332836 7fcdffcdb700 0 log [INF] : pgmap v29082: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3236B/s wr, 0op/s -10826> 2013-04-03 10:42:58.332838 7fcdffcdb700 10 send_log to self -10825> 2013-04-03 10:42:58.332839 7fcdffcdb700 10 log_queue is 2 last_log 148 sent 146 num 2 unsent 2 sending 2 -10824> 2013-04-03 10:42:58.332841 7fcdffcdb700 10 will send 2013-04-03 10:42:52.721369 mon.0 10.252.24.25:6789/0 147 : [INF] pgmap v29081: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -10823> 2013-04-03 10:42:58.332846 7fcdffcdb700 10 will send 2013-04-03 10:42:58.332837 mon.0 10.252.24.25:6789/0 148 : [INF] pgmap v29082: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3236B/s wr, 0op/s -10822> 2013-04-03 10:42:58.332851 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b73e00 -10821> 2013-04-03 10:42:58.332898 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos -10820> 2013-04-03 10:42:58.332918 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 -10819> 2013-04-03 10:42:58.332937 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10818> 2013-04-03 10:42:58.332962 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73e00 con 0x1bac160 -10817> 2013-04-03 10:42:58.332985 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos -10816> 2013-04-03 10:42:58.333000 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 -10815> 2013-04-03 10:42:58.333010 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -10814> 2013-04-03 10:42:58.333021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 preprocess_log log(2 entries) v1 from mon.0 -10813> 2013-04-03 10:42:58.333053 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -10812> 2013-04-03 10:42:58.333066 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 prepare_log log(2 entries) v1 from mon.0 -10811> 2013-04-03 10:42:58.333076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 logging 2013-04-03 10:42:52.721369 mon.0 10.252.24.25:6789/0 147 : [INF] pgmap v29081: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -10810> 2013-04-03 10:42:58.333097 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 logging 2013-04-03 10:42:58.332837 mon.0 10.252.24.25:6789/0 148 : [INF] pgmap v29082: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3236B/s wr, 0op/s -10809> 2013-04-03 10:42:58.356319 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113763 ==== paxos(lease_ack lc 427446 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (2567935141 0 0) 0x2df2a00 con 0x208a840 -10808> 2013-04-03 10:42:58.356354 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos -10807> 2013-04-03 10:42:58.356371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 -10806> 2013-04-03 10:42:58.356387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10805> 2013-04-03 10:42:58.363662 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 check_down_pgs -10804> 2013-04-03 10:42:58.363688 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29082 v29082: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3236B/s wr, 0op/s -10803> 2013-04-03 10:42:58.363716 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -10802> 2013-04-03 10:42:58.363741 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -10801> 2013-04-03 10:42:58.363779 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -10800> 2013-04-03 10:42:58.363810 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos -10799> 2013-04-03 10:42:58.363829 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 update_from_paxos version 394077 summary v 394077 -10798> 2013-04-03 10:42:58.363838 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394077 log -10797> 2013-04-03 10:42:58.363855 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10796> 2013-04-03 10:42:58.363874 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -10795> 2013-04-03 10:42:58.694978 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) -10794> 2013-04-03 10:42:58.695018 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader -10793> 2013-04-03 10:42:59.332487 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394077 encode_pending v394078 -10792> 2013-04-03 10:42:59.332516 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427446) queue_proposal bl 725 bytes; ctx = 0x2c3d8f0 -10791> 2013-04-03 10:42:59.332524 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427446) propose_queued 427447 725 bytes -10790> 2013-04-03 10:42:59.333612 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427446 fc 0 pn 17600 opn 0) v3 -- ?+0 0x1f91c80 -10789> 2013-04-03 10:42:59.355750 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113764 ==== paxos(accept lc 427446 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4192077425 0 0) 0x1f91c80 con 0x208a840 -10788> 2013-04-03 10:42:59.356079 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427447 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2a00 -10787> 2013-04-03 10:42:59.356111 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427447 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2df2780 -10786> 2013-04-03 10:42:59.356149 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos -10785> 2013-04-03 10:42:59.356173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394077 -10784> 2013-04-03 10:42:59.356187 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos latest full 393798 -10783> 2013-04-03 10:42:59.356209 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394078 update_from_paxos applying incremental log 394078 2013-04-03 10:42:52.721369 mon.0 10.252.24.25:6789/0 147 : [INF] pgmap v29081: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -10782> 2013-04-03 10:42:59.356237 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394078 update_from_paxos applying incremental log 394078 2013-04-03 10:42:58.332837 mon.0 10.252.24.25:6789/0 148 : [INF] pgmap v29082: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3236B/s wr, 0op/s -10781> 2013-04-03 10:42:59.356273 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 check_subs -10780> 2013-04-03 10:42:59.356304 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 create_pending v 394079 -10779> 2013-04-03 10:42:59.356326 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394078 _updated_log for mon.0 10.252.24.25:6789/0 -10778> 2013-04-03 10:42:59.356339 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 148) v1 -- ?+0 0x28a5a80 con 0x1bac160 -10777> 2013-04-03 10:42:59.356391 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos -10776> 2013-04-03 10:42:59.356406 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 -10775> 2013-04-03 10:42:59.356423 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10774> 2013-04-03 10:42:59.356445 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 148) v1 ==== 0+0+0 (0 0 0) 0x28a5a80 con 0x1bac160 -10773> 2013-04-03 10:42:59.356453 7fcdffcdb700 10 handle_log_ack log(last 148) v1 -10772> 2013-04-03 10:42:59.356454 7fcdffcdb700 10 logged 2013-04-03 10:42:52.721369 mon.0 10.252.24.25:6789/0 147 : [INF] pgmap v29081: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -10771> 2013-04-03 10:42:59.356461 7fcdffcdb700 10 logged 2013-04-03 10:42:58.332837 mon.0 10.252.24.25:6789/0 148 : [INF] pgmap v29082: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3236B/s wr, 0op/s -10770> 2013-04-03 10:42:59.368842 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113765 ==== paxos(lease_ack lc 427447 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (2784209801 0 0) 0x2df2780 con 0x208a840 -10769> 2013-04-03 10:42:59.368910 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos -10768> 2013-04-03 10:42:59.368929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 -10767> 2013-04-03 10:42:59.368946 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10766> 2013-04-03 10:42:59.592246 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 143 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140428 v187) v2 ==== 289+0+0 (734693366 0 0) 0x2f79840 con 0x2118420 -10765> 2013-04-03 10:42:59.592322 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140428 v187) v2 from mds.? 10.252.58.31:6800/10908 -10764> 2013-04-03 10:42:59.592338 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140428 v187) v2 -- ?+0 0x2f79b00 con 0x2118420 -10763> 2013-04-03 10:43:00.938685 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113766 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 143 v187) v2) to leader v1 ==== 560+0+0 (4244355553 0 0) 0x2cc5b80 con 0x208a840 -10762> 2013-04-03 10:43:00.938707 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -10761> 2013-04-03 10:43:00.938716 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2146080 from 10.252.24.88:6789/0 -10760> 2013-04-03 10:43:00.938769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 143 v187) v2 from mds.0 10.252.24.25:6802/3144 -10759> 2013-04-03 10:43:00.938780 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 143 v187) v2 tid 17660) v2 -- ?+0 0x24f8d80 con 0x208a840 -10758> 2013-04-03 10:43:01.390735 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 143 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21799 v187) v2 ==== 289+0+0 (891963154 0 0) 0x2efa340 con 0x200d420 -10757> 2013-04-03 10:43:01.390813 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21799 v187) v2 from mds.? 10.252.24.88:6800/17213 -10756> 2013-04-03 10:43:01.390826 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21799 v187) v2 -- ?+0 0x2cc5b80 con 0x200d420 -10755> 2013-04-03 10:43:02.356186 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427447 fc 427429 pn 0 opn 0) v3 -- ?+0 0x20f4780 -10754> 2013-04-03 10:43:02.356950 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113767 ==== paxos(lease_ack lc 427447 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (794222940 0 0) 0x20f4780 con 0x208a840 -10753> 2013-04-03 10:43:02.357017 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos -10752> 2013-04-03 10:43:02.357037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 -10751> 2013-04-03 10:43:02.357054 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10750> 2013-04-03 10:43:03.364012 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29082 v29082: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 3236B/s wr, 0op/s -10749> 2013-04-03 10:43:03.364059 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -10748> 2013-04-03 10:43:03.364080 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -10747> 2013-04-03 10:43:03.364124 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -10746> 2013-04-03 10:43:03.364156 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos -10745> 2013-04-03 10:43:03.364174 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 -10744> 2013-04-03 10:43:03.364184 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394078 log -10743> 2013-04-03 10:43:03.364200 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10742> 2013-04-03 10:43:03.364219 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -10741> 2013-04-03 10:43:03.592339 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 144 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140429 v187) v2 ==== 289+0+0 (144980107 0 0) 0x2f79580 con 0x2118420 -10740> 2013-04-03 10:43:03.592419 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140429 v187) v2 from mds.? 10.252.58.31:6800/10908 -10739> 2013-04-03 10:43:03.592432 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140429 v187) v2 -- ?+0 0x2efa340 con 0x2118420 -10738> 2013-04-03 10:43:04.938663 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113768 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 144 v187) v2) to leader v1 ==== 560+0+0 (214933259 0 0) 0x2f79840 con 0x208a840 -10737> 2013-04-03 10:43:04.938683 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -10736> 2013-04-03 10:43:04.938693 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2145840 from 10.252.24.88:6789/0 -10735> 2013-04-03 10:43:04.938750 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 144 v187) v2 from mds.0 10.252.24.25:6802/3144 -10734> 2013-04-03 10:43:04.938762 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 144 v187) v2 tid 17661) v2 -- ?+0 0x24f8900 con 0x208a840 -10733> 2013-04-03 10:43:05.356254 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427447 fc 427429 pn 0 opn 0) v3 -- ?+0 0x20f4280 -10732> 2013-04-03 10:43:05.356951 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113769 ==== paxos(lease_ack lc 427447 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (76805437 0 0) 0x20f4280 con 0x208a840 -10731> 2013-04-03 10:43:05.357015 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos -10730> 2013-04-03 10:43:05.357035 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 -10729> 2013-04-03 10:43:05.357052 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10728> 2013-04-03 10:43:05.390846 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 144 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21800 v187) v2 ==== 289+0+0 (3283809912 0 0) 0x2efa8c0 con 0x200d420 -10727> 2013-04-03 10:43:05.390917 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21800 v187) v2 from mds.? 10.252.24.88:6800/17213 -10726> 2013-04-03 10:43:05.390930 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21800 v187) v2 -- ?+0 0x2f79840 con 0x200d420 -10725> 2013-04-03 10:43:06.613417 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113770 ==== forward(pg_stats(1 pgs tid 45 v 605) v1) to leader v1 ==== 753+0+0 (4007907974 0 0) 0x2f79580 con 0x208a840 -10724> 2013-04-03 10:43:06.613436 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -10723> 2013-04-03 10:43:06.613446 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -10722> 2013-04-03 10:43:06.613498 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29082 preprocess_query pg_stats(1 pgs tid 45 v 605) v1 from osd.1 10.252.24.25:6805/3572 -10721> 2013-04-03 10:43:06.613523 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29082 prepare_update pg_stats(1 pgs tid 45 v 605) v1 from osd.1 10.252.24.25:6805/3572 -10720> 2013-04-03 10:43:06.613529 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29082 prepare_pg_stats pg_stats(1 pgs tid 45 v 605) v1 from osd.1 -10719> 2013-04-03 10:43:06.613535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29082 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -10718> 2013-04-03 10:43:06.663720 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29082 encode_pending v 29083 -10717> 2013-04-03 10:43:06.663756 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427447) queue_proposal bl 650 bytes; ctx = 0x2c3d8e0 -10716> 2013-04-03 10:43:06.663765 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427447) propose_queued 427448 650 bytes -10715> 2013-04-03 10:43:06.664840 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427447 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4a00 -10714> 2013-04-03 10:43:06.692390 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113771 ==== paxos(accept lc 427447 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1305595556 0 0) 0x20f4a00 con 0x208a840 -10713> 2013-04-03 10:43:06.692736 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427448 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4280 -10712> 2013-04-03 10:43:06.692765 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427448 fc 427429 pn 0 opn 0) v3 -- ?+0 0x20f4780 -10711> 2013-04-03 10:43:06.692840 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29082 update_from_paxos applying incremental 29083 -10710> 2013-04-03 10:43:06.692855 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 v29083: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s -10709> 2013-04-03 10:43:06.692875 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 send_pg_creates to 0 pgs -10708> 2013-04-03 10:43:06.692877 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 update_logger -10707> 2013-04-03 10:43:06.692886 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 create_pending v 29084 -10706> 2013-04-03 10:43:06.692895 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29083 _updated_stats for osd.1 10.252.24.25:6805/3572 -10705> 2013-04-03 10:43:06.692902 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 45) v1 tid 17662) v2 -- ?+0 0x24f8d80 con 0x208a840 -10704> 2013-04-03 10:43:06.692911 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 check_osd_map already seen 605 >= 605 -10703> 2013-04-03 10:43:06.692914 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 update_logger -10702> 2013-04-03 10:43:06.692924 7fcdffcdb700 0 log [INF] : pgmap v29083: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s -10701> 2013-04-03 10:43:06.692926 7fcdffcdb700 10 send_log to self -10700> 2013-04-03 10:43:06.692927 7fcdffcdb700 10 log_queue is 1 last_log 149 sent 148 num 1 unsent 1 sending 1 -10699> 2013-04-03 10:43:06.692928 7fcdffcdb700 10 will send 2013-04-03 10:43:06.692925 mon.0 10.252.24.25:6789/0 149 : [INF] pgmap v29083: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s -10698> 2013-04-03 10:43:06.692934 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73e00 -10697> 2013-04-03 10:43:06.692985 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos -10696> 2013-04-03 10:43:06.693004 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 -10695> 2013-04-03 10:43:06.693025 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10694> 2013-04-03 10:43:06.693046 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73e00 con 0x1bac160 -10693> 2013-04-03 10:43:06.693068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos -10692> 2013-04-03 10:43:06.693085 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 -10691> 2013-04-03 10:43:06.693094 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10690> 2013-04-03 10:43:06.693106 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 preprocess_log log(1 entries) v1 from mon.0 -10689> 2013-04-03 10:43:06.693130 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10688> 2013-04-03 10:43:06.693141 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 prepare_log log(1 entries) v1 from mon.0 -10687> 2013-04-03 10:43:06.693151 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 logging 2013-04-03 10:43:06.692925 mon.0 10.252.24.25:6789/0 149 : [INF] pgmap v29083: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s -10686> 2013-04-03 10:43:06.718599 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113772 ==== paxos(lease_ack lc 427448 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (1032458762 0 0) 0x20f4780 con 0x208a840 -10685> 2013-04-03 10:43:06.718642 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos -10684> 2013-04-03 10:43:06.718659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394078 update_from_paxos version 394078 summary v 394078 -10683> 2013-04-03 10:43:06.718674 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10682> 2013-04-03 10:43:07.592435 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 145 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140430 v187) v2 ==== 289+0+0 (1830287628 0 0) 0x2f792c0 con 0x2118420 -10681> 2013-04-03 10:43:07.592508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140430 v187) v2 from mds.? 10.252.58.31:6800/10908 -10680> 2013-04-03 10:43:07.592520 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140430 v187) v2 -- ?+0 0x2f79580 con 0x2118420 -10679> 2013-04-03 10:43:07.692577 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394078 encode_pending v394079 -10678> 2013-04-03 10:43:07.692602 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427448) queue_proposal bl 443 bytes; ctx = 0x2c3d800 -10677> 2013-04-03 10:43:07.692609 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427448) propose_queued 427449 443 bytes -10676> 2013-04-03 10:43:07.694064 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427448 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4c80 -10675> 2013-04-03 10:43:07.720790 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113773 ==== paxos(accept lc 427448 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3181164535 0 0) 0x20f4c80 con 0x208a840 -10674> 2013-04-03 10:43:07.721121 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427449 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4780 -10673> 2013-04-03 10:43:07.721154 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427449 fc 427429 pn 0 opn 0) v3 -- ?+0 0x20f4a00 -10672> 2013-04-03 10:43:07.721189 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos -10671> 2013-04-03 10:43:07.721214 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394078 -10670> 2013-04-03 10:43:07.721229 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos latest full 393798 -10669> 2013-04-03 10:43:07.721248 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394079 update_from_paxos applying incremental log 394079 2013-04-03 10:43:06.692925 mon.0 10.252.24.25:6789/0 149 : [INF] pgmap v29083: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s -10668> 2013-04-03 10:43:07.721295 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 check_subs -10667> 2013-04-03 10:43:07.721326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 create_pending v 394080 -10666> 2013-04-03 10:43:07.721349 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394079 _updated_log for mon.0 10.252.24.25:6789/0 -10665> 2013-04-03 10:43:07.721364 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 149) v1 -- ?+0 0x28a5a80 con 0x1bac160 -10664> 2013-04-03 10:43:07.721413 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos -10663> 2013-04-03 10:43:07.721429 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 -10662> 2013-04-03 10:43:07.721447 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10661> 2013-04-03 10:43:07.721468 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 149) v1 ==== 0+0+0 (0 0 0) 0x28a5a80 con 0x1bac160 -10660> 2013-04-03 10:43:07.721476 7fcdffcdb700 10 handle_log_ack log(last 149) v1 -10659> 2013-04-03 10:43:07.721477 7fcdffcdb700 10 logged 2013-04-03 10:43:06.692925 mon.0 10.252.24.25:6789/0 149 : [INF] pgmap v29083: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s -10658> 2013-04-03 10:43:07.733155 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113774 ==== paxos(lease_ack lc 427449 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (4039783150 0 0) 0x20f4a00 con 0x208a840 -10657> 2013-04-03 10:43:07.733191 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos -10656> 2013-04-03 10:43:07.733207 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 -10655> 2013-04-03 10:43:07.733224 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10654> 2013-04-03 10:43:08.364336 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29083 check_down_pgs -10653> 2013-04-03 10:43:08.364372 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29083 v29083: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1758B/s wr, 0op/s -10652> 2013-04-03 10:43:08.364400 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -10651> 2013-04-03 10:43:08.364420 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -10650> 2013-04-03 10:43:08.364456 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -10649> 2013-04-03 10:43:08.364486 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos -10648> 2013-04-03 10:43:08.364502 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 -10647> 2013-04-03 10:43:08.364511 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394079 log -10646> 2013-04-03 10:43:08.364527 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10645> 2013-04-03 10:43:08.364544 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -10644> 2013-04-03 10:43:08.938808 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113775 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 145 v187) v2) to leader v1 ==== 560+0+0 (3518667358 0 0) 0x2145580 con 0x208a840 -10643> 2013-04-03 10:43:08.938830 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -10642> 2013-04-03 10:43:08.938840 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae340 from 10.252.24.88:6789/0 -10641> 2013-04-03 10:43:08.938892 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 145 v187) v2 from mds.0 10.252.24.25:6802/3144 -10640> 2013-04-03 10:43:08.938905 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 145 v187) v2 tid 17663) v2 -- ?+0 0x24f8900 con 0x208a840 -10639> 2013-04-03 10:43:09.391011 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 145 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21801 v187) v2 ==== 289+0+0 (3772008709 0 0) 0x2a51000 con 0x200d420 -10638> 2013-04-03 10:43:09.391086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21801 v187) v2 from mds.? 10.252.24.88:6800/17213 -10637> 2013-04-03 10:43:09.391099 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21801 v187) v2 -- ?+0 0x2145580 con 0x200d420 -10636> 2013-04-03 10:43:10.721231 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427449 fc 427429 pn 0 opn 0) v3 -- ?+0 0x20f4500 -10635> 2013-04-03 10:43:10.722016 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113776 ==== paxos(lease_ack lc 427449 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (43948895 0 0) 0x20f4500 con 0x208a840 -10634> 2013-04-03 10:43:10.722084 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos -10633> 2013-04-03 10:43:10.722104 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 -10632> 2013-04-03 10:43:10.722122 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10631> 2013-04-03 10:43:10.937153 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2df2000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33103/0 -10630> 2013-04-03 10:43:10.937173 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33103/0) -10629> 2013-04-03 10:43:10.937186 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2000 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -10628> 2013-04-03 10:43:10.937400 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2000 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -10627> 2013-04-03 10:43:11.592581 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 146 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140431 v187) v2 ==== 289+0+0 (1316570737 0 0) 0x2f79000 con 0x2118420 -10626> 2013-04-03 10:43:11.592660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140431 v187) v2 from mds.? 10.252.58.31:6800/10908 -10625> 2013-04-03 10:43:11.592673 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140431 v187) v2 -- ?+0 0x2a51000 con 0x2118420 -10624> 2013-04-03 10:43:12.939002 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113777 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 146 v187) v2) to leader v1 ==== 560+0+0 (1176304821 0 0) 0x2f792c0 con 0x208a840 -10623> 2013-04-03 10:43:12.939021 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -10622> 2013-04-03 10:43:12.939030 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae600 from 10.252.24.88:6789/0 -10621> 2013-04-03 10:43:12.939084 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 146 v187) v2 from mds.0 10.252.24.25:6802/3144 -10620> 2013-04-03 10:43:12.939094 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 146 v187) v2 tid 17664) v2 -- ?+0 0x24f8d80 con 0x208a840 -10619> 2013-04-03 10:43:13.258623 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113778 ==== forward(pg_stats(1 pgs tid 26 v 605) v1) to leader v1 ==== 757+0+0 (4027269839 0 0) 0x2f79000 con 0x208a840 -10618> 2013-04-03 10:43:13.258637 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -10617> 2013-04-03 10:43:13.258644 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -10616> 2013-04-03 10:43:13.258680 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 preprocess_query pg_stats(1 pgs tid 26 v 605) v1 from osd.2 10.252.24.30:6801/5656 -10615> 2013-04-03 10:43:13.258704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 prepare_update pg_stats(1 pgs tid 26 v 605) v1 from osd.2 10.252.24.30:6801/5656 -10614> 2013-04-03 10:43:13.258710 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 prepare_pg_stats pg_stats(1 pgs tid 26 v 605) v1 from osd.2 -10613> 2013-04-03 10:43:13.258716 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29083 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -10612> 2013-04-03 10:43:13.308898 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29083 encode_pending v 29084 -10611> 2013-04-03 10:43:13.308932 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427449) queue_proposal bl 654 bytes; ctx = 0x2c3d8d0 -10610> 2013-04-03 10:43:13.308941 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427449) propose_queued 427450 654 bytes -10609> 2013-04-03 10:43:13.310073 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427449 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4780 -10608> 2013-04-03 10:43:13.339395 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113779 ==== paxos(accept lc 427449 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (930893355 0 0) 0x29b4780 con 0x208a840 -10607> 2013-04-03 10:43:13.339752 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427450 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4500 -10606> 2013-04-03 10:43:13.339779 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427450 fc 427429 pn 0 opn 0) v3 -- ?+0 0x20f4a00 -10605> 2013-04-03 10:43:13.339853 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29083 update_from_paxos applying incremental 29084 -10604> 2013-04-03 10:43:13.339869 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 v29084: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -10603> 2013-04-03 10:43:13.339888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 send_pg_creates to 0 pgs -10602> 2013-04-03 10:43:13.339892 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 update_logger -10601> 2013-04-03 10:43:13.339902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 create_pending v 29085 -10600> 2013-04-03 10:43:13.339910 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29084 _updated_stats for osd.2 10.252.24.30:6801/5656 -10599> 2013-04-03 10:43:13.339917 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 26) v1 tid 17665) v2 -- ?+0 0x24f8900 con 0x208a840 -10598> 2013-04-03 10:43:13.339926 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 check_osd_map already seen 605 >= 605 -10597> 2013-04-03 10:43:13.339931 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 update_logger -10596> 2013-04-03 10:43:13.339941 7fcdffcdb700 0 log [INF] : pgmap v29084: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -10595> 2013-04-03 10:43:13.339943 7fcdffcdb700 10 send_log to self -10594> 2013-04-03 10:43:13.339943 7fcdffcdb700 10 log_queue is 1 last_log 150 sent 149 num 1 unsent 1 sending 1 -10593> 2013-04-03 10:43:13.339945 7fcdffcdb700 10 will send 2013-04-03 10:43:13.339942 mon.0 10.252.24.25:6789/0 150 : [INF] pgmap v29084: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -10592> 2013-04-03 10:43:13.339951 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73e00 -10591> 2013-04-03 10:43:13.339998 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos -10590> 2013-04-03 10:43:13.340017 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 -10589> 2013-04-03 10:43:13.340036 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10588> 2013-04-03 10:43:13.340059 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73e00 con 0x1bac160 -10587> 2013-04-03 10:43:13.340081 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos -10586> 2013-04-03 10:43:13.340096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 -10585> 2013-04-03 10:43:13.340104 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10584> 2013-04-03 10:43:13.340114 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 preprocess_log log(1 entries) v1 from mon.0 -10583> 2013-04-03 10:43:13.340136 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10582> 2013-04-03 10:43:13.340148 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 prepare_log log(1 entries) v1 from mon.0 -10581> 2013-04-03 10:43:13.340157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 logging 2013-04-03 10:43:13.339942 mon.0 10.252.24.25:6789/0 150 : [INF] pgmap v29084: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -10580> 2013-04-03 10:43:13.364655 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29084 check_down_pgs -10579> 2013-04-03 10:43:13.364684 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29084 v29084: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -10578> 2013-04-03 10:43:13.364710 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -10577> 2013-04-03 10:43:13.364730 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -10576> 2013-04-03 10:43:13.364766 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -10575> 2013-04-03 10:43:13.364797 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos -10574> 2013-04-03 10:43:13.364816 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 -10573> 2013-04-03 10:43:13.364826 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394079 log -10572> 2013-04-03 10:43:13.364841 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10571> 2013-04-03 10:43:13.364858 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -10570> 2013-04-03 10:43:13.365518 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113780 ==== paxos(lease_ack lc 427450 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3425151703 0 0) 0x20f4a00 con 0x208a840 -10569> 2013-04-03 10:43:13.365553 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos -10568> 2013-04-03 10:43:13.365571 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394079 update_from_paxos version 394079 summary v 394079 -10567> 2013-04-03 10:43:13.365588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10566> 2013-04-03 10:43:13.391114 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 146 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21802 v187) v2 ==== 289+0+0 (2238154882 0 0) 0x2a52080 con 0x200d420 -10565> 2013-04-03 10:43:13.391190 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21802 v187) v2 from mds.? 10.252.24.88:6800/17213 -10564> 2013-04-03 10:43:13.391203 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21802 v187) v2 -- ?+0 0x2f79000 con 0x200d420 -10563> 2013-04-03 10:43:14.339946 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394079 encode_pending v394080 -10562> 2013-04-03 10:43:14.339975 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427450) queue_proposal bl 443 bytes; ctx = 0x2c3d7f0 -10561> 2013-04-03 10:43:14.339982 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427450) propose_queued 427451 443 bytes -10560> 2013-04-03 10:43:14.341077 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427450 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4280 -10559> 2013-04-03 10:43:14.365575 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113781 ==== paxos(accept lc 427450 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3477373190 0 0) 0x29b4280 con 0x208a840 -10558> 2013-04-03 10:43:14.365889 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427451 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4a00 -10557> 2013-04-03 10:43:14.365918 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427451 fc 427429 pn 0 opn 0) v3 -- ?+0 0x29b4780 -10556> 2013-04-03 10:43:14.365957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos -10555> 2013-04-03 10:43:14.365985 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394079 -10554> 2013-04-03 10:43:14.366002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos latest full 393798 -10553> 2013-04-03 10:43:14.366021 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394080 update_from_paxos applying incremental log 394080 2013-04-03 10:43:13.339942 mon.0 10.252.24.25:6789/0 150 : [INF] pgmap v29084: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -10552> 2013-04-03 10:43:14.366066 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 check_subs -10551> 2013-04-03 10:43:14.366098 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 create_pending v 394081 -10550> 2013-04-03 10:43:14.366120 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394080 _updated_log for mon.0 10.252.24.25:6789/0 -10549> 2013-04-03 10:43:14.366132 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 150) v1 -- ?+0 0x28a58c0 con 0x1bac160 -10548> 2013-04-03 10:43:14.366186 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos -10547> 2013-04-03 10:43:14.366203 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 -10546> 2013-04-03 10:43:14.366221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10545> 2013-04-03 10:43:14.366245 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 150) v1 ==== 0+0+0 (0 0 0) 0x28a58c0 con 0x1bac160 -10544> 2013-04-03 10:43:14.366254 7fcdffcdb700 10 handle_log_ack log(last 150) v1 -10543> 2013-04-03 10:43:14.366256 7fcdffcdb700 10 logged 2013-04-03 10:43:13.339942 mon.0 10.252.24.25:6789/0 150 : [INF] pgmap v29084: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -10542> 2013-04-03 10:43:14.377886 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113782 ==== paxos(lease_ack lc 427451 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (2414133144 0 0) 0x29b4780 con 0x208a840 -10541> 2013-04-03 10:43:14.377922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos -10540> 2013-04-03 10:43:14.377938 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 -10539> 2013-04-03 10:43:14.377954 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10538> 2013-04-03 10:43:15.592711 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 147 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140432 v187) v2 ==== 289+0+0 (2245717289 0 0) 0x20e4340 con 0x2118420 -10537> 2013-04-03 10:43:15.592790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140432 v187) v2 from mds.? 10.252.58.31:6800/10908 -10536> 2013-04-03 10:43:15.592803 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140432 v187) v2 -- ?+0 0x2a52080 con 0x2118420 -10535> 2013-04-03 10:43:16.939072 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113783 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 147 v187) v2) to leader v1 ==== 560+0+0 (2607413728 0 0) 0x2ce1dc0 con 0x208a840 -10534> 2013-04-03 10:43:16.939092 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -10533> 2013-04-03 10:43:16.939102 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2145dc0 from 10.252.24.88:6789/0 -10532> 2013-04-03 10:43:16.939154 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 147 v187) v2 from mds.0 10.252.24.25:6802/3144 -10531> 2013-04-03 10:43:16.939166 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 147 v187) v2 tid 17666) v2 -- ?+0 0x24f8d80 con 0x208a840 -10530> 2013-04-03 10:43:17.365989 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427451 fc 427429 pn 0 opn 0) v3 -- ?+0 0x29b4a00 -10529> 2013-04-03 10:43:17.366737 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113784 ==== paxos(lease_ack lc 427451 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (1264703142 0 0) 0x29b4a00 con 0x208a840 -10528> 2013-04-03 10:43:17.366784 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos -10527> 2013-04-03 10:43:17.366803 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 -10526> 2013-04-03 10:43:17.366820 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10525> 2013-04-03 10:43:17.391181 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 147 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21803 v187) v2 ==== 289+0+0 (2785622015 0 0) 0x2a512c0 con 0x200d420 -10524> 2013-04-03 10:43:17.391255 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21803 v187) v2 from mds.? 10.252.24.88:6800/17213 -10523> 2013-04-03 10:43:17.391268 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21803 v187) v2 -- ?+0 0x2ce1dc0 con 0x200d420 -10522> 2013-04-03 10:43:18.364995 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29084 v29084: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -10521> 2013-04-03 10:43:18.365038 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -10520> 2013-04-03 10:43:18.365060 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -10519> 2013-04-03 10:43:18.365104 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -10518> 2013-04-03 10:43:18.365135 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos -10517> 2013-04-03 10:43:18.365154 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 -10516> 2013-04-03 10:43:18.365163 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394080 log -10515> 2013-04-03 10:43:18.365180 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10514> 2013-04-03 10:43:18.365199 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -10513> 2013-04-03 10:43:19.592809 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 148 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140433 v187) v2 ==== 289+0+0 (2796835412 0 0) 0x20e3840 con 0x2118420 -10512> 2013-04-03 10:43:19.592897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140433 v187) v2 from mds.? 10.252.58.31:6800/10908 -10511> 2013-04-03 10:43:19.592911 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140433 v187) v2 -- ?+0 0x2a512c0 con 0x2118420 -10510> 2013-04-03 10:43:20.366062 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427451 fc 427429 pn 0 opn 0) v3 -- ?+0 0x29b4000 -10509> 2013-04-03 10:43:20.366879 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113785 ==== paxos(lease_ack lc 427451 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3932848082 0 0) 0x29b4000 con 0x208a840 -10508> 2013-04-03 10:43:20.366946 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos -10507> 2013-04-03 10:43:20.366966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 -10506> 2013-04-03 10:43:20.366984 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10505> 2013-04-03 10:43:20.939157 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113786 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 148 v187) v2) to leader v1 ==== 560+0+0 (134292404 0 0) 0x20e4340 con 0x208a840 -10504> 2013-04-03 10:43:20.939176 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -10503> 2013-04-03 10:43:20.939186 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x21452c0 from 10.252.24.88:6789/0 -10502> 2013-04-03 10:43:20.939239 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 148 v187) v2 from mds.0 10.252.24.25:6802/3144 -10501> 2013-04-03 10:43:20.939250 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 148 v187) v2 tid 17667) v2 -- ?+0 0x24f8900 con 0x208a840 -10500> 2013-04-03 10:43:21.391291 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 148 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21804 v187) v2 ==== 289+0+0 (1308625804 0 0) 0x2a52600 con 0x200d420 -10499> 2013-04-03 10:43:21.391363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21804 v187) v2 from mds.? 10.252.24.88:6800/17213 -10498> 2013-04-03 10:43:21.391376 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21804 v187) v2 -- ?+0 0x20e4340 con 0x200d420 -10497> 2013-04-03 10:43:21.615050 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113787 ==== forward(pg_stats(1 pgs tid 46 v 605) v1) to leader v1 ==== 753+0+0 (3031482359 0 0) 0x20e3840 con 0x208a840 -10496> 2013-04-03 10:43:21.615072 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -10495> 2013-04-03 10:43:21.615082 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -10494> 2013-04-03 10:43:21.615134 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 preprocess_query pg_stats(1 pgs tid 46 v 605) v1 from osd.1 10.252.24.25:6805/3572 -10493> 2013-04-03 10:43:21.615158 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 prepare_update pg_stats(1 pgs tid 46 v 605) v1 from osd.1 10.252.24.25:6805/3572 -10492> 2013-04-03 10:43:21.615164 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 prepare_pg_stats pg_stats(1 pgs tid 46 v 605) v1 from osd.1 -10491> 2013-04-03 10:43:21.615170 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29084 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -10490> 2013-04-03 10:43:21.665376 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29084 encode_pending v 29085 -10489> 2013-04-03 10:43:21.665417 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427451) queue_proposal bl 650 bytes; ctx = 0x2c3d8b0 -10488> 2013-04-03 10:43:21.665428 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427451) propose_queued 427452 650 bytes -10487> 2013-04-03 10:43:21.666367 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427451 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2000 -10486> 2013-04-03 10:43:21.694954 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113788 ==== paxos(accept lc 427451 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3991774138 0 0) 0x2df2000 con 0x208a840 -10485> 2013-04-03 10:43:21.695860 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427452 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4000 -10484> 2013-04-03 10:43:21.695889 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427452 fc 427429 pn 0 opn 0) v3 -- ?+0 0x29b4a00 -10483> 2013-04-03 10:43:21.695962 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29084 update_from_paxos applying incremental 29085 -10482> 2013-04-03 10:43:21.695981 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 v29085: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -10481> 2013-04-03 10:43:21.696000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 send_pg_creates to 0 pgs -10480> 2013-04-03 10:43:21.696003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 update_logger -10479> 2013-04-03 10:43:21.696012 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 create_pending v 29086 -10478> 2013-04-03 10:43:21.696021 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29085 _updated_stats for osd.1 10.252.24.25:6805/3572 -10477> 2013-04-03 10:43:21.696028 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 46) v1 tid 17668) v2 -- ?+0 0x24f8d80 con 0x208a840 -10476> 2013-04-03 10:43:21.696038 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 check_osd_map already seen 605 >= 605 -10475> 2013-04-03 10:43:21.696040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 update_logger -10474> 2013-04-03 10:43:21.696050 7fcdffcdb700 0 log [INF] : pgmap v29085: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -10473> 2013-04-03 10:43:21.696058 7fcdffcdb700 10 send_log to self -10472> 2013-04-03 10:43:21.696058 7fcdffcdb700 10 log_queue is 1 last_log 151 sent 150 num 1 unsent 1 sending 1 -10471> 2013-04-03 10:43:21.696060 7fcdffcdb700 10 will send 2013-04-03 10:43:21.696057 mon.0 10.252.24.25:6789/0 151 : [INF] pgmap v29085: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -10470> 2013-04-03 10:43:21.696066 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73e00 -10469> 2013-04-03 10:43:21.696119 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos -10468> 2013-04-03 10:43:21.696138 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 -10467> 2013-04-03 10:43:21.696157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10466> 2013-04-03 10:43:21.696178 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73e00 con 0x1bac160 -10465> 2013-04-03 10:43:21.696200 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos -10464> 2013-04-03 10:43:21.696215 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 -10463> 2013-04-03 10:43:21.696224 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10462> 2013-04-03 10:43:21.696236 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 preprocess_log log(1 entries) v1 from mon.0 -10461> 2013-04-03 10:43:21.696260 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10460> 2013-04-03 10:43:21.696271 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 prepare_log log(1 entries) v1 from mon.0 -10459> 2013-04-03 10:43:21.696281 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 logging 2013-04-03 10:43:21.696057 mon.0 10.252.24.25:6789/0 151 : [INF] pgmap v29085: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -10458> 2013-04-03 10:43:21.708638 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113789 ==== paxos(lease_ack lc 427452 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3794849622 0 0) 0x29b4a00 con 0x208a840 -10457> 2013-04-03 10:43:21.708686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos -10456> 2013-04-03 10:43:21.708704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394080 update_from_paxos version 394080 summary v 394080 -10455> 2013-04-03 10:43:21.708719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10454> 2013-04-03 10:43:21.780745 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113790 ==== forward(pg_stats(1 pgs tid 30455 v 605) v1) to leader v1 ==== 757+0+0 (2802694182 0 0) 0x2ce2080 con 0x208a840 -10453> 2013-04-03 10:43:21.780759 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -10452> 2013-04-03 10:43:21.780766 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -10451> 2013-04-03 10:43:21.780795 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 preprocess_query pg_stats(1 pgs tid 30455 v 605) v1 from osd.3 10.252.24.88:6801/17538 -10450> 2013-04-03 10:43:21.780816 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 prepare_update pg_stats(1 pgs tid 30455 v 605) v1 from osd.3 10.252.24.88:6801/17538 -10449> 2013-04-03 10:43:21.780821 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 prepare_pg_stats pg_stats(1 pgs tid 30455 v 605) v1 from osd.3 -10448> 2013-04-03 10:43:21.780827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29085 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -10447> 2013-04-03 10:43:22.695161 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394080 encode_pending v394081 -10446> 2013-04-03 10:43:22.695188 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427452) queue_proposal bl 443 bytes; ctx = 0x2c3d7e0 -10445> 2013-04-03 10:43:22.695195 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427452) propose_queued 427453 443 bytes -10444> 2013-04-03 10:43:22.696317 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427452 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4c80 -10443> 2013-04-03 10:43:22.696404 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29085 encode_pending v 29086 -10442> 2013-04-03 10:43:22.696439 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427429..427452) queue_proposal bl 654 bytes; ctx = 0x2c3d7d0 -10441> 2013-04-03 10:43:22.696446 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427429..427452) propose_new_value not active; proposal queued -10440> 2013-04-03 10:43:22.725374 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113791 ==== paxos(accept lc 427452 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1461547965 0 0) 0x2fe4c80 con 0x208a840 -10439> 2013-04-03 10:43:22.725671 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427453 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4a00 -10438> 2013-04-03 10:43:22.725697 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427453 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2df2000 -10437> 2013-04-03 10:43:22.725740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos -10436> 2013-04-03 10:43:22.725766 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos version 394081 summary v 394080 -10435> 2013-04-03 10:43:22.725781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos latest full 393798 -10434> 2013-04-03 10:43:22.725799 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394081 update_from_paxos applying incremental log 394081 2013-04-03 10:43:21.696057 mon.0 10.252.24.25:6789/0 151 : [INF] pgmap v29085: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -10433> 2013-04-03 10:43:22.725856 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 check_subs -10432> 2013-04-03 10:43:22.725889 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 create_pending v 394082 -10431> 2013-04-03 10:43:22.725911 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394081 _updated_log for mon.0 10.252.24.25:6789/0 -10430> 2013-04-03 10:43:22.725923 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 151) v1 -- ?+0 0x28a5540 con 0x1bac160 -10429> 2013-04-03 10:43:22.725947 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427453) propose_queued 427454 654 bytes -10428> 2013-04-03 10:43:22.726158 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427453 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4780 -10427> 2013-04-03 10:43:22.726192 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 151) v1 ==== 0+0+0 (0 0 0) 0x28a5540 con 0x1bac160 -10426> 2013-04-03 10:43:22.726208 7fcdffcdb700 10 handle_log_ack log(last 151) v1 -10425> 2013-04-03 10:43:22.726210 7fcdffcdb700 10 logged 2013-04-03 10:43:21.696057 mon.0 10.252.24.25:6789/0 151 : [INF] pgmap v29085: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -10424> 2013-04-03 10:43:22.762848 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113792 ==== paxos(lease_ack lc 427453 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (2979370495 0 0) 0x29b4780 con 0x208a840 -10423> 2013-04-03 10:43:22.910303 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113793 ==== paxos(accept lc 427453 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (575171680 0 0) 0x2df2000 con 0x208a840 -10422> 2013-04-03 10:43:22.910599 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427454 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4780 -10421> 2013-04-03 10:43:22.910631 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427454 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2fe4c80 -10420> 2013-04-03 10:43:22.910699 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29085 update_from_paxos applying incremental 29086 -10419> 2013-04-03 10:43:22.910718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 v29086: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -10418> 2013-04-03 10:43:22.910736 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 send_pg_creates to 0 pgs -10417> 2013-04-03 10:43:22.910740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 update_logger -10416> 2013-04-03 10:43:22.910750 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 create_pending v 29087 -10415> 2013-04-03 10:43:22.910759 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29086 _updated_stats for osd.3 10.252.24.88:6801/17538 -10414> 2013-04-03 10:43:22.910765 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30455) v1 tid 17669) v2 -- ?+0 0x24f8900 con 0x208a840 -10413> 2013-04-03 10:43:22.910775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 check_osd_map already seen 605 >= 605 -10412> 2013-04-03 10:43:22.910778 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 update_logger -10411> 2013-04-03 10:43:22.910788 7fcdffcdb700 0 log [INF] : pgmap v29086: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -10410> 2013-04-03 10:43:22.910789 7fcdffcdb700 10 send_log to self -10409> 2013-04-03 10:43:22.910795 7fcdffcdb700 10 log_queue is 1 last_log 152 sent 151 num 1 unsent 1 sending 1 -10408> 2013-04-03 10:43:22.910797 7fcdffcdb700 10 will send 2013-04-03 10:43:22.910789 mon.0 10.252.24.25:6789/0 152 : [INF] pgmap v29086: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -10407> 2013-04-03 10:43:22.910804 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b78400 -10406> 2013-04-03 10:43:22.910855 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos -10405> 2013-04-03 10:43:22.910872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos version 394081 summary v 394081 -10404> 2013-04-03 10:43:22.910891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10403> 2013-04-03 10:43:22.910911 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b78400 con 0x1bac160 -10402> 2013-04-03 10:43:22.910933 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos -10401> 2013-04-03 10:43:22.910947 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos version 394081 summary v 394081 -10400> 2013-04-03 10:43:22.910956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10399> 2013-04-03 10:43:22.910968 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 preprocess_log log(1 entries) v1 from mon.0 -10398> 2013-04-03 10:43:22.910988 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10397> 2013-04-03 10:43:22.911000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 prepare_log log(1 entries) v1 from mon.0 -10396> 2013-04-03 10:43:22.911011 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 logging 2013-04-03 10:43:22.910789 mon.0 10.252.24.25:6789/0 152 : [INF] pgmap v29086: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -10395> 2013-04-03 10:43:22.997894 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113794 ==== paxos(lease_ack lc 427454 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (887764295 0 0) 0x2fe4c80 con 0x208a840 -10394> 2013-04-03 10:43:22.997936 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos -10393> 2013-04-03 10:43:22.997952 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos version 394081 summary v 394081 -10392> 2013-04-03 10:43:22.997967 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10391> 2013-04-03 10:43:23.365321 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29086 check_down_pgs -10390> 2013-04-03 10:43:23.365351 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29086 v29086: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -10389> 2013-04-03 10:43:23.365377 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -10388> 2013-04-03 10:43:23.365398 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -10387> 2013-04-03 10:43:23.365435 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -10386> 2013-04-03 10:43:23.365465 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos -10385> 2013-04-03 10:43:23.365482 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394081 update_from_paxos version 394081 summary v 394081 -10384> 2013-04-03 10:43:23.365491 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394081 log -10383> 2013-04-03 10:43:23.365511 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10382> 2013-04-03 10:43:23.365528 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -10381> 2013-04-03 10:43:23.592918 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 149 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140434 v187) v2 ==== 289+0+0 (3272053715 0 0) 0x1baeb80 con 0x2118420 -10380> 2013-04-03 10:43:23.592999 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140434 v187) v2 from mds.? 10.252.58.31:6800/10908 -10379> 2013-04-03 10:43:23.593013 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140434 v187) v2 -- ?+0 0x2ce2080 con 0x2118420 -10378> 2013-04-03 10:43:23.910504 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394081 encode_pending v394082 -10377> 2013-04-03 10:43:23.910537 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427454) queue_proposal bl 442 bytes; ctx = 0x2c3d8c0 -10376> 2013-04-03 10:43:23.910545 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427454) propose_queued 427455 442 bytes -10375> 2013-04-03 10:43:23.911810 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427454 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4a00 -10374> 2013-04-03 10:43:23.943030 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113795 ==== paxos(accept lc 427454 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4078995512 0 0) 0x2fe4a00 con 0x208a840 -10373> 2013-04-03 10:43:23.943326 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427455 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4c80 -10372> 2013-04-03 10:43:23.943359 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427455 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2df2000 -10371> 2013-04-03 10:43:23.943403 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos -10370> 2013-04-03 10:43:23.943428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394081 -10369> 2013-04-03 10:43:23.943443 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos latest full 393798 -10368> 2013-04-03 10:43:23.943461 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394082 update_from_paxos applying incremental log 394082 2013-04-03 10:43:22.910789 mon.0 10.252.24.25:6789/0 152 : [INF] pgmap v29086: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -10367> 2013-04-03 10:43:23.943507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 check_subs -10366> 2013-04-03 10:43:23.943538 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 create_pending v 394083 -10365> 2013-04-03 10:43:23.943560 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394082 _updated_log for mon.0 10.252.24.25:6789/0 -10364> 2013-04-03 10:43:23.943573 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 152) v1 -- ?+0 0x28a5540 con 0x1bac160 -10363> 2013-04-03 10:43:23.943624 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos -10362> 2013-04-03 10:43:23.943640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394082 -10361> 2013-04-03 10:43:23.943658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10360> 2013-04-03 10:43:23.943681 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 152) v1 ==== 0+0+0 (0 0 0) 0x28a5540 con 0x1bac160 -10359> 2013-04-03 10:43:23.943689 7fcdffcdb700 10 handle_log_ack log(last 152) v1 -10358> 2013-04-03 10:43:23.943690 7fcdffcdb700 10 logged 2013-04-03 10:43:22.910789 mon.0 10.252.24.25:6789/0 152 : [INF] pgmap v29086: 240 pgs: 240 active+clean; 255 MB data, 5658 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -10357> 2013-04-03 10:43:23.969084 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113796 ==== paxos(lease_ack lc 427455 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3903312958 0 0) 0x2df2000 con 0x208a840 -10356> 2013-04-03 10:43:23.969124 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos -10355> 2013-04-03 10:43:23.969141 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394082 -10354> 2013-04-03 10:43:23.969157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10353> 2013-04-03 10:43:24.939300 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113797 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 149 v187) v2) to leader v1 ==== 560+0+0 (888516494 0 0) 0x2145b00 con 0x208a840 -10352> 2013-04-03 10:43:24.939322 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -10351> 2013-04-03 10:43:24.939334 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e3b00 from 10.252.24.88:6789/0 -10350> 2013-04-03 10:43:24.939386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 149 v187) v2 from mds.0 10.252.24.25:6802/3144 -10349> 2013-04-03 10:43:24.939398 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 149 v187) v2 tid 17670) v2 -- ?+0 0x24f8d80 con 0x208a840 -10348> 2013-04-03 10:43:25.391350 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 149 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21805 v187) v2 ==== 289+0+0 (1835971825 0 0) 0x2a51b00 con 0x200d420 -10347> 2013-04-03 10:43:25.391402 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21805 v187) v2 from mds.? 10.252.24.88:6800/17213 -10346> 2013-04-03 10:43:25.391412 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21805 v187) v2 -- ?+0 0x2145b00 con 0x200d420 -10345> 2013-04-03 10:43:26.943436 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427455 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2fe4780 -10344> 2013-04-03 10:43:26.944175 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113798 ==== paxos(lease_ack lc 427455 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (2910424430 0 0) 0x2fe4780 con 0x208a840 -10343> 2013-04-03 10:43:26.944242 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos -10342> 2013-04-03 10:43:26.944262 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394082 -10341> 2013-04-03 10:43:26.944279 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10340> 2013-04-03 10:43:27.593039 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 150 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140435 v187) v2 ==== 289+0+0 (3764989102 0 0) 0x1bae8c0 con 0x2118420 -10339> 2013-04-03 10:43:27.593117 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140435 v187) v2 from mds.? 10.252.58.31:6800/10908 -10338> 2013-04-03 10:43:27.593130 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140435 v187) v2 -- ?+0 0x2a51b00 con 0x2118420 -10337> 2013-04-03 10:43:28.260763 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113799 ==== forward(pg_stats(1 pgs tid 27 v 605) v1) to leader v1 ==== 757+0+0 (2059652380 0 0) 0x1baeb80 con 0x208a840 -10336> 2013-04-03 10:43:28.260783 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -10335> 2013-04-03 10:43:28.260792 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -10334> 2013-04-03 10:43:28.260842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 preprocess_query pg_stats(1 pgs tid 27 v 605) v1 from osd.2 10.252.24.30:6801/5656 -10333> 2013-04-03 10:43:28.260867 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 prepare_update pg_stats(1 pgs tid 27 v 605) v1 from osd.2 10.252.24.30:6801/5656 -10332> 2013-04-03 10:43:28.260873 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 prepare_pg_stats pg_stats(1 pgs tid 27 v 605) v1 from osd.2 -10331> 2013-04-03 10:43:28.260878 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29086 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -10330> 2013-04-03 10:43:28.311063 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29086 encode_pending v 29087 -10329> 2013-04-03 10:43:28.311101 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427455) queue_proposal bl 654 bytes; ctx = 0x2c3d6b0 -10328> 2013-04-03 10:43:28.311110 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427455) propose_queued 427456 654 bytes -10327> 2013-04-03 10:43:28.313004 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427455 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4500 -10326> 2013-04-03 10:43:28.365642 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -10325> 2013-04-03 10:43:28.365683 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -10324> 2013-04-03 10:43:28.365906 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos -10323> 2013-04-03 10:43:28.365929 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394082 -10322> 2013-04-03 10:43:28.365938 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394082 log -10321> 2013-04-03 10:43:28.365956 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10320> 2013-04-03 10:43:28.365975 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -10319> 2013-04-03 10:43:28.507502 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113800 ==== paxos(accept lc 427455 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3937658362 0 0) 0x2fe4500 con 0x208a840 -10318> 2013-04-03 10:43:28.507892 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427456 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4780 -10317> 2013-04-03 10:43:28.507914 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427456 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2df2000 -10316> 2013-04-03 10:43:28.508000 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29086 update_from_paxos applying incremental 29087 -10315> 2013-04-03 10:43:28.508022 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29087 v29087: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -10314> 2013-04-03 10:43:28.508040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29087 send_pg_creates to 0 pgs -10313> 2013-04-03 10:43:28.508044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29087 update_logger -10312> 2013-04-03 10:43:28.508055 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29087 create_pending v 29088 -10311> 2013-04-03 10:43:28.508064 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29087 _updated_stats for osd.2 10.252.24.30:6801/5656 -10310> 2013-04-03 10:43:28.508071 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 27) v1 tid 17671) v2 -- ?+0 0x24f8900 con 0x208a840 -10309> 2013-04-03 10:43:28.508082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29087 check_osd_map already seen 605 >= 605 -10308> 2013-04-03 10:43:28.508085 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29087 update_logger -10307> 2013-04-03 10:43:28.508102 7fcdffcdb700 0 log [INF] : pgmap v29087: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -10306> 2013-04-03 10:43:28.508105 7fcdffcdb700 10 send_log to self -10305> 2013-04-03 10:43:28.508106 7fcdffcdb700 10 log_queue is 1 last_log 153 sent 152 num 1 unsent 1 sending 1 -10304> 2013-04-03 10:43:28.508107 7fcdffcdb700 10 will send 2013-04-03 10:43:28.508104 mon.0 10.252.24.25:6789/0 153 : [INF] pgmap v29087: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -10303> 2013-04-03 10:43:28.508114 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b78400 -10302> 2013-04-03 10:43:28.508167 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos -10301> 2013-04-03 10:43:28.508186 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394082 -10300> 2013-04-03 10:43:28.508204 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10299> 2013-04-03 10:43:28.508225 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b78400 con 0x1bac160 -10298> 2013-04-03 10:43:28.508247 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos -10297> 2013-04-03 10:43:28.508262 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394082 -10296> 2013-04-03 10:43:28.508271 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10295> 2013-04-03 10:43:28.508282 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 preprocess_log log(1 entries) v1 from mon.0 -10294> 2013-04-03 10:43:28.508307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10293> 2013-04-03 10:43:28.508320 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 prepare_log log(1 entries) v1 from mon.0 -10292> 2013-04-03 10:43:28.508330 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 logging 2013-04-03 10:43:28.508104 mon.0 10.252.24.25:6789/0 153 : [INF] pgmap v29087: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -10291> 2013-04-03 10:43:28.519791 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113801 ==== paxos(lease_ack lc 427456 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (2689053137 0 0) 0x2df2000 con 0x208a840 -10290> 2013-04-03 10:43:28.519828 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos -10289> 2013-04-03 10:43:28.519844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394082 update_from_paxos version 394082 summary v 394082 -10288> 2013-04-03 10:43:28.519862 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -10287> 2013-04-03 10:43:28.759351 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113802 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (424405211 0 0) 0x273a000 con 0x208a840 -10286> 2013-04-03 10:43:28.759379 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -10285> 2013-04-03 10:43:28.759386 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -10284> 2013-04-03 10:43:28.759395 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 -10283> 2013-04-03 10:43:28.759399 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( start_reply flags( reply-to(mon.2 10.252.24.88:6789/0) ) ) v1 -- ?+0 0x1baeb80 -10282> 2013-04-03 10:43:28.939426 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113803 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 150 v187) v2) to leader v1 ==== 560+0+0 (1039954674 0 0) 0x273b8c0 con 0x208a840 -10281> 2013-04-03 10:43:28.939441 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -10280> 2013-04-03 10:43:28.939450 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20e4600 from 10.252.24.88:6789/0 -10279> 2013-04-03 10:43:28.939503 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 150 v187) v2 from mds.0 10.252.24.25:6802/3144 -10278> 2013-04-03 10:43:28.939515 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 150 v187) v2 tid 17672) v2 -- ?+0 0x24f8d80 con 0x208a840 -10277> 2013-04-03 10:43:29.391488 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 150 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21806 v187) v2 ==== 289+0+0 (148731254 0 0) 0x2cc4840 con 0x200d420 -10276> 2013-04-03 10:43:29.391537 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21806 v187) v2 from mds.? 10.252.24.88:6800/17213 -10275> 2013-04-03 10:43:29.391547 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21806 v187) v2 -- ?+0 0x273b8c0 con 0x200d420 -10274> 2013-04-03 10:43:29.507725 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394082 encode_pending v394083 -10273> 2013-04-03 10:43:29.507758 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427456) queue_proposal bl 443 bytes; ctx = 0x2c3d820 -10272> 2013-04-03 10:43:29.507766 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427456) propose_queued 427457 443 bytes -10271> 2013-04-03 10:43:29.508856 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427456 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4280 -10270> 2013-04-03 10:43:29.537626 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113804 ==== paxos(accept lc 427456 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1669527613 0 0) 0x2fe4280 con 0x208a840 -10269> 2013-04-03 10:43:29.537921 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427457 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2000 -10268> 2013-04-03 10:43:29.537943 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427457 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2fe4500 -10267> 2013-04-03 10:43:29.537978 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos -10266> 2013-04-03 10:43:29.538011 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394082 -10265> 2013-04-03 10:43:29.538029 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos latest full 393798 -10264> 2013-04-03 10:43:29.538049 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos applying incremental log 394083 2013-04-03 10:43:28.508104 mon.0 10.252.24.25:6789/0 153 : [INF] pgmap v29087: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -10263> 2013-04-03 10:43:29.538095 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 check_subs -10262> 2013-04-03 10:43:29.538126 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 create_pending v 394084 -10261> 2013-04-03 10:43:29.538149 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 _updated_log for mon.0 10.252.24.25:6789/0 -10260> 2013-04-03 10:43:29.538163 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 153) v1 -- ?+0 0x2224380 con 0x1bac160 -10259> 2013-04-03 10:43:29.538215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos -10258> 2013-04-03 10:43:29.538231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 -10257> 2013-04-03 10:43:29.538250 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10256> 2013-04-03 10:43:29.538274 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 153) v1 ==== 0+0+0 (0 0 0) 0x2224380 con 0x1bac160 -10255> 2013-04-03 10:43:29.538282 7fcdffcdb700 10 handle_log_ack log(last 153) v1 -10254> 2013-04-03 10:43:29.538283 7fcdffcdb700 10 logged 2013-04-03 10:43:28.508104 mon.0 10.252.24.25:6789/0 153 : [INF] pgmap v29087: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -10253> 2013-04-03 10:43:29.563721 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113805 ==== paxos(lease_ack lc 427457 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3059861107 0 0) 0x2fe4500 con 0x208a840 -10252> 2013-04-03 10:43:29.563763 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos -10251> 2013-04-03 10:43:29.563780 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 -10250> 2013-04-03 10:43:29.563798 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10249> 2013-04-03 10:43:30.969200 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3003c80 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33110/0 -10248> 2013-04-03 10:43:30.969227 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003c80 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33110/0) -10247> 2013-04-03 10:43:30.969243 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003c80 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -10246> 2013-04-03 10:43:30.969446 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003c80 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -10245> 2013-04-03 10:43:31.593200 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 151 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140436 v187) v2 ==== 289+0+0 (140513501 0 0) 0x2ef9dc0 con 0x2118420 -10244> 2013-04-03 10:43:31.593278 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140436 v187) v2 from mds.? 10.252.58.31:6800/10908 -10243> 2013-04-03 10:43:31.593292 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140436 v187) v2 -- ?+0 0x2cc4840 con 0x2118420 -10242> 2013-04-03 10:43:32.538020 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427457 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2fe4000 -10241> 2013-04-03 10:43:32.538834 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113806 ==== paxos(lease_ack lc 427457 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (907135810 0 0) 0x2fe4000 con 0x208a840 -10240> 2013-04-03 10:43:32.538909 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos -10239> 2013-04-03 10:43:32.538930 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 -10238> 2013-04-03 10:43:32.538948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10237> 2013-04-03 10:43:32.939674 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113807 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 151 v187) v2) to leader v1 ==== 560+0+0 (884321521 0 0) 0x273a000 con 0x208a840 -10236> 2013-04-03 10:43:32.939696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -10235> 2013-04-03 10:43:32.939707 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x20e3dc0 from 10.252.24.88:6789/0 -10234> 2013-04-03 10:43:32.939753 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 151 v187) v2 from mds.0 10.252.24.25:6802/3144 -10233> 2013-04-03 10:43:32.939766 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 151 v187) v2 tid 17673) v2 -- ?+0 0x24f8900 con 0x208a840 -10232> 2013-04-03 10:43:33.366097 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 check_down_pgs -10231> 2013-04-03 10:43:33.366131 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 v29087: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2003B/s wr, 0op/s -10230> 2013-04-03 10:43:33.366157 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -10229> 2013-04-03 10:43:33.366181 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -10228> 2013-04-03 10:43:33.366218 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -10227> 2013-04-03 10:43:33.366250 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos -10226> 2013-04-03 10:43:33.366270 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 -10225> 2013-04-03 10:43:33.366279 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 log -10224> 2013-04-03 10:43:33.366296 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10223> 2013-04-03 10:43:33.366316 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -10222> 2013-04-03 10:43:33.391604 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 151 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21807 v187) v2 ==== 289+0+0 (733202955 0 0) 0x2cc5600 con 0x200d420 -10221> 2013-04-03 10:43:33.391653 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21807 v187) v2 from mds.? 10.252.24.88:6800/17213 -10220> 2013-04-03 10:43:33.391665 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21807 v187) v2 -- ?+0 0x273a000 con 0x200d420 -10219> 2013-04-03 10:43:35.507660 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113808 ==== forward(pg_stats(0 pgs tid 62 v 605) v1) to leader v1 ==== 368+0+0 (642871096 0 0) 0x2ef9dc0 con 0x208a840 -10218> 2013-04-03 10:43:35.507688 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 -10217> 2013-04-03 10:43:35.507700 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8900 from 10.252.24.88:6789/0 -10216> 2013-04-03 10:43:35.507754 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 preprocess_query pg_stats(0 pgs tid 62 v 605) v1 from osd.4 10.252.24.32:6800/13473 -10215> 2013-04-03 10:43:35.507779 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 prepare_update pg_stats(0 pgs tid 62 v 605) v1 from osd.4 10.252.24.32:6800/13473 -10214> 2013-04-03 10:43:35.507786 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 prepare_pg_stats pg_stats(0 pgs tid 62 v 605) v1 from osd.4 -10213> 2013-04-03 10:43:35.507792 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 got osd.4 osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[])) -10212> 2013-04-03 10:43:35.538086 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427457 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3003c80 -10211> 2013-04-03 10:43:35.538784 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113809 ==== paxos(lease_ack lc 427457 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (1525944739 0 0) 0x3003c80 con 0x208a840 -10210> 2013-04-03 10:43:35.538839 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos -10209> 2013-04-03 10:43:35.538860 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 -10208> 2013-04-03 10:43:35.538878 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10207> 2013-04-03 10:43:35.557972 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 encode_pending v 29088 -10206> 2013-04-03 10:43:35.558008 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427457) queue_proposal bl 265 bytes; ctx = 0x2c3d810 -10205> 2013-04-03 10:43:35.558018 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427457) propose_queued 427458 265 bytes -10204> 2013-04-03 10:43:35.558793 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427457 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300cc80 -10203> 2013-04-03 10:43:35.584867 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113810 ==== paxos(accept lc 427457 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1001114336 0 0) 0x300cc80 con 0x208a840 -10202> 2013-04-03 10:43:35.585852 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427458 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003c80 -10201> 2013-04-03 10:43:35.585879 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427458 fc 427429 pn 0 opn 0) v3 -- ?+0 0x2fe4000 -10200> 2013-04-03 10:43:35.585960 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29087 update_from_paxos applying incremental 29088 -10199> 2013-04-03 10:43:35.585972 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 v29088: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 955B/s wr, 0op/s -10198> 2013-04-03 10:43:35.585990 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 send_pg_creates to 0 pgs -10197> 2013-04-03 10:43:35.585993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 update_logger -10196> 2013-04-03 10:43:35.586003 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 create_pending v 29089 -10195> 2013-04-03 10:43:35.586015 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 _updated_stats for osd.4 10.252.24.32:6800/13473 -10194> 2013-04-03 10:43:35.586022 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(0 pgs tid 62) v1 tid 17674) v2 -- ?+0 0x24f8d80 con 0x208a840 -10193> 2013-04-03 10:43:35.586032 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 check_osd_map already seen 605 >= 605 -10192> 2013-04-03 10:43:35.586038 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 update_logger -10191> 2013-04-03 10:43:35.586048 7fcdffcdb700 0 log [INF] : pgmap v29088: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 955B/s wr, 0op/s -10190> 2013-04-03 10:43:35.586050 7fcdffcdb700 10 send_log to self -10189> 2013-04-03 10:43:35.586051 7fcdffcdb700 10 log_queue is 1 last_log 154 sent 153 num 1 unsent 1 sending 1 -10188> 2013-04-03 10:43:35.586052 7fcdffcdb700 10 will send 2013-04-03 10:43:35.586049 mon.0 10.252.24.25:6789/0 154 : [INF] pgmap v29088: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 955B/s wr, 0op/s -10187> 2013-04-03 10:43:35.586059 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b78400 -10186> 2013-04-03 10:43:35.586106 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos -10185> 2013-04-03 10:43:35.586125 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 -10184> 2013-04-03 10:43:35.586144 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10183> 2013-04-03 10:43:35.586164 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b78400 con 0x1bac160 -10182> 2013-04-03 10:43:35.586186 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos -10181> 2013-04-03 10:43:35.586201 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 -10180> 2013-04-03 10:43:35.586211 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10179> 2013-04-03 10:43:35.586223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 preprocess_log log(1 entries) v1 from mon.0 -10178> 2013-04-03 10:43:35.586248 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10177> 2013-04-03 10:43:35.586262 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 prepare_log log(1 entries) v1 from mon.0 -10176> 2013-04-03 10:43:35.586274 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 logging 2013-04-03 10:43:35.586049 mon.0 10.252.24.25:6789/0 154 : [INF] pgmap v29088: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 955B/s wr, 0op/s -10175> 2013-04-03 10:43:35.593295 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 152 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140437 v187) v2 ==== 289+0+0 (722382752 0 0) 0x2ef9b00 con 0x2118420 -10174> 2013-04-03 10:43:35.593333 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140437 v187) v2 from mds.? 10.252.58.31:6800/10908 -10173> 2013-04-03 10:43:35.593346 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140437 v187) v2 -- ?+0 0x2ef9dc0 con 0x2118420 -10172> 2013-04-03 10:43:35.598532 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113811 ==== paxos(lease_ack lc 427458 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (4242275755 0 0) 0x2fe4000 con 0x208a840 -10171> 2013-04-03 10:43:35.598567 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos -10170> 2013-04-03 10:43:35.598584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 update_from_paxos version 394083 summary v 394083 -10169> 2013-04-03 10:43:35.598600 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10168> 2013-04-03 10:43:36.585077 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394083 encode_pending v394084 -10167> 2013-04-03 10:43:36.585106 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427458) queue_proposal bl 442 bytes; ctx = 0x2c3d7b0 -10166> 2013-04-03 10:43:36.585114 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427458) propose_queued 427459 442 bytes -10165> 2013-04-03 10:43:36.586215 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427458 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300ca00 -10164> 2013-04-03 10:43:36.615338 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113812 ==== paxos(accept lc 427458 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1862058324 0 0) 0x300ca00 con 0x208a840 -10163> 2013-04-03 10:43:36.615631 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427459 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4000 -10162> 2013-04-03 10:43:36.615648 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427459 fc 427429 pn 0 opn 0) v3 -- ?+0 0x300cc80 -10161> 2013-04-03 10:43:36.615681 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos -10160> 2013-04-03 10:43:36.615709 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos version 394084 summary v 394083 -10159> 2013-04-03 10:43:36.615725 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos latest full 393798 -10158> 2013-04-03 10:43:36.615744 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos applying incremental log 394084 2013-04-03 10:43:35.586049 mon.0 10.252.24.25:6789/0 154 : [INF] pgmap v29088: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 955B/s wr, 0op/s -10157> 2013-04-03 10:43:36.615792 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 check_subs -10156> 2013-04-03 10:43:36.615824 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 create_pending v 394085 -10155> 2013-04-03 10:43:36.615847 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 _updated_log for mon.0 10.252.24.25:6789/0 -10154> 2013-04-03 10:43:36.615861 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 154) v1 -- ?+0 0x2224700 con 0x1bac160 -10153> 2013-04-03 10:43:36.615912 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos -10152> 2013-04-03 10:43:36.616139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos version 394084 summary v 394084 -10151> 2013-04-03 10:43:36.616159 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10150> 2013-04-03 10:43:36.616184 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 154) v1 ==== 0+0+0 (0 0 0) 0x2224700 con 0x1bac160 -10149> 2013-04-03 10:43:36.616193 7fcdffcdb700 10 handle_log_ack log(last 154) v1 -10148> 2013-04-03 10:43:36.616194 7fcdffcdb700 10 logged 2013-04-03 10:43:35.586049 mon.0 10.252.24.25:6789/0 154 : [INF] pgmap v29088: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 955B/s wr, 0op/s -10147> 2013-04-03 10:43:36.627755 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113813 ==== paxos(lease_ack lc 427459 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (507044061 0 0) 0x300cc80 con 0x208a840 -10146> 2013-04-03 10:43:36.627791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos -10145> 2013-04-03 10:43:36.627808 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos version 394084 summary v 394084 -10144> 2013-04-03 10:43:36.627824 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10143> 2013-04-03 10:43:36.627996 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113814 ==== forward(pg_stats(1 pgs tid 47 v 605) v1) to leader v1 ==== 753+0+0 (1579122226 0 0) 0x2cc4000 con 0x208a840 -10142> 2013-04-03 10:43:36.628008 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -10141> 2013-04-03 10:43:36.628016 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -10140> 2013-04-03 10:43:36.628046 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 preprocess_query pg_stats(1 pgs tid 47 v 605) v1 from osd.1 10.252.24.25:6805/3572 -10139> 2013-04-03 10:43:36.628068 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 prepare_update pg_stats(1 pgs tid 47 v 605) v1 from osd.1 10.252.24.25:6805/3572 -10138> 2013-04-03 10:43:36.628074 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 prepare_pg_stats pg_stats(1 pgs tid 47 v 605) v1 from osd.1 -10137> 2013-04-03 10:43:36.628081 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -10136> 2013-04-03 10:43:36.842976 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113815 ==== forward(pg_stats(0 pgs tid 411 v 605) v1) to leader v1 ==== 384+0+0 (2315706269 0 0) 0x2cc4b00 con 0x208a840 -10135> 2013-04-03 10:43:36.843003 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -10134> 2013-04-03 10:43:36.843016 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -10133> 2013-04-03 10:43:36.843065 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 preprocess_query pg_stats(0 pgs tid 411 v 605) v1 from osd.0 10.252.58.31:6802/2682 -10132> 2013-04-03 10:43:36.843090 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 prepare_update pg_stats(0 pgs tid 411 v 605) v1 from osd.0 10.252.58.31:6802/2682 -10131> 2013-04-03 10:43:36.843103 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 prepare_pg_stats pg_stats(0 pgs tid 411 v 605) v1 from osd.0 -10130> 2013-04-03 10:43:36.843110 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 got osd.0 osd_stat(1141 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1140 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -10129> 2013-04-03 10:43:36.939695 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113816 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 152 v187) v2) to leader v1 ==== 560+0+0 (4282399172 0 0) 0x2a83b00 con 0x208a840 -10128> 2013-04-03 10:43:36.939709 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -10127> 2013-04-03 10:43:36.939717 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83580 from 10.252.24.88:6789/0 -10126> 2013-04-03 10:43:36.939751 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 152 v187) v2 from mds.0 10.252.24.25:6802/3144 -10125> 2013-04-03 10:43:36.939764 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 152 v187) v2 tid 17677) v2 -- ?+0 0x24f8900 con 0x208a840 -10124> 2013-04-03 10:43:37.391823 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 152 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21808 v187) v2 ==== 289+0+0 (3759169875 0 0) 0x2a83840 con 0x200d420 -10123> 2013-04-03 10:43:37.391875 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21808 v187) v2 from mds.? 10.252.24.88:6800/17213 -10122> 2013-04-03 10:43:37.391885 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21808 v187) v2 -- ?+0 0x2a83b00 con 0x200d420 -10121> 2013-04-03 10:43:37.615538 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 encode_pending v 29089 -10120> 2013-04-03 10:43:37.615574 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427459) queue_proposal bl 716 bytes; ctx = 0x2c3d8a0 -10119> 2013-04-03 10:43:37.615584 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427459) propose_queued 427460 716 bytes -10118> 2013-04-03 10:43:37.617350 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427459 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300c780 -10117> 2013-04-03 10:43:37.736112 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113817 ==== paxos(accept lc 427459 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3118953454 0 0) 0x300c780 con 0x208a840 -10116> 2013-04-03 10:43:37.736451 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427460 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300cc80 -10115> 2013-04-03 10:43:37.736478 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427460 fc 427429 pn 0 opn 0) v3 -- ?+0 0x300ca00 -10114> 2013-04-03 10:43:37.736548 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29088 update_from_paxos applying incremental 29089 -10113> 2013-04-03 10:43:37.736570 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 v29089: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 440B/s wr, 0op/s -10112> 2013-04-03 10:43:37.736591 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 send_pg_creates to 0 pgs -10111> 2013-04-03 10:43:37.736595 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 update_logger -10110> 2013-04-03 10:43:37.736606 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 create_pending v 29090 -10109> 2013-04-03 10:43:37.736616 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 _updated_stats for osd.1 10.252.24.25:6805/3572 -10108> 2013-04-03 10:43:37.736624 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 47) v1 tid 17675) v2 -- ?+0 0x24f8240 con 0x208a840 -10107> 2013-04-03 10:43:37.736635 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 _updated_stats for osd.0 10.252.58.31:6802/2682 -10106> 2013-04-03 10:43:37.736639 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(0 pgs tid 411) v1 tid 17676) v2 -- ?+0 0x24f8d80 con 0x208a840 -10105> 2013-04-03 10:43:37.736669 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 check_osd_map already seen 605 >= 605 -10104> 2013-04-03 10:43:37.736673 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 update_logger -10103> 2013-04-03 10:43:37.736684 7fcdffcdb700 0 log [INF] : pgmap v29089: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 440B/s wr, 0op/s -10102> 2013-04-03 10:43:37.736686 7fcdffcdb700 10 send_log to self -10101> 2013-04-03 10:43:37.736687 7fcdffcdb700 10 log_queue is 1 last_log 155 sent 154 num 1 unsent 1 sending 1 -10100> 2013-04-03 10:43:37.736688 7fcdffcdb700 10 will send 2013-04-03 10:43:37.736685 mon.0 10.252.24.25:6789/0 155 : [INF] pgmap v29089: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 440B/s wr, 0op/s -10099> 2013-04-03 10:43:37.736696 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b78200 -10098> 2013-04-03 10:43:37.736748 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos -10097> 2013-04-03 10:43:37.736768 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos version 394084 summary v 394084 -10096> 2013-04-03 10:43:37.736787 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10095> 2013-04-03 10:43:37.736809 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b78200 con 0x1bac160 -10094> 2013-04-03 10:43:37.736831 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos -10093> 2013-04-03 10:43:37.736846 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos version 394084 summary v 394084 -10092> 2013-04-03 10:43:37.736856 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10091> 2013-04-03 10:43:37.736868 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 preprocess_log log(1 entries) v1 from mon.0 -10090> 2013-04-03 10:43:37.736894 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -10089> 2013-04-03 10:43:37.736909 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 prepare_log log(1 entries) v1 from mon.0 -10088> 2013-04-03 10:43:37.736921 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 logging 2013-04-03 10:43:37.736685 mon.0 10.252.24.25:6789/0 155 : [INF] pgmap v29089: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 440B/s wr, 0op/s -10087> 2013-04-03 10:43:37.762307 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113818 ==== paxos(lease_ack lc 427460 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3097400084 0 0) 0x300ca00 con 0x208a840 -10086> 2013-04-03 10:43:37.762342 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos -10085> 2013-04-03 10:43:37.762358 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos version 394084 summary v 394084 -10084> 2013-04-03 10:43:37.762373 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10083> 2013-04-03 10:43:38.366465 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 check_down_pgs -10082> 2013-04-03 10:43:38.366492 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 v29089: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 440B/s wr, 0op/s -10081> 2013-04-03 10:43:38.366520 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -10080> 2013-04-03 10:43:38.366545 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -10079> 2013-04-03 10:43:38.366582 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -10078> 2013-04-03 10:43:38.366614 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos -10077> 2013-04-03 10:43:38.366633 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 update_from_paxos version 394084 summary v 394084 -10076> 2013-04-03 10:43:38.366643 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 log -10075> 2013-04-03 10:43:38.366660 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10074> 2013-04-03 10:43:38.366679 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -10073> 2013-04-03 10:43:38.736318 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394084 encode_pending v394085 -10072> 2013-04-03 10:43:38.736350 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427460) queue_proposal bl 442 bytes; ctx = 0x2c3d890 -10071> 2013-04-03 10:43:38.736358 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427460) propose_queued 427461 442 bytes -10070> 2013-04-03 10:43:38.737441 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427460 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300c500 -10069> 2013-04-03 10:43:38.762240 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113819 ==== paxos(accept lc 427460 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (638351688 0 0) 0x300c500 con 0x208a840 -10068> 2013-04-03 10:43:38.762563 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427461 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300ca00 -10067> 2013-04-03 10:43:38.762594 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427461 fc 427429 pn 0 opn 0) v3 -- ?+0 0x300c780 -10066> 2013-04-03 10:43:38.762627 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos -10065> 2013-04-03 10:43:38.762655 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394084 -10064> 2013-04-03 10:43:38.762670 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos latest full 393798 -10063> 2013-04-03 10:43:38.763169 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos applying incremental log 394085 2013-04-03 10:43:37.736685 mon.0 10.252.24.25:6789/0 155 : [INF] pgmap v29089: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 440B/s wr, 0op/s -10062> 2013-04-03 10:43:38.763226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 check_subs -10061> 2013-04-03 10:43:38.763259 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 create_pending v 394086 -10060> 2013-04-03 10:43:38.763282 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 _updated_log for mon.0 10.252.24.25:6789/0 -10059> 2013-04-03 10:43:38.763296 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 155) v1 -- ?+0 0x22248c0 con 0x1bac160 -10058> 2013-04-03 10:43:38.763347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos -10057> 2013-04-03 10:43:38.763364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394085 -10056> 2013-04-03 10:43:38.763385 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10055> 2013-04-03 10:43:38.763408 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 155) v1 ==== 0+0+0 (0 0 0) 0x22248c0 con 0x1bac160 -10054> 2013-04-03 10:43:38.763416 7fcdffcdb700 10 handle_log_ack log(last 155) v1 -10053> 2013-04-03 10:43:38.763417 7fcdffcdb700 10 logged 2013-04-03 10:43:37.736685 mon.0 10.252.24.25:6789/0 155 : [INF] pgmap v29089: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 440B/s wr, 0op/s -10052> 2013-04-03 10:43:38.774645 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113820 ==== paxos(lease_ack lc 427461 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3520981979 0 0) 0x300c780 con 0x208a840 -10051> 2013-04-03 10:43:38.774680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos -10050> 2013-04-03 10:43:38.774698 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394085 -10049> 2013-04-03 10:43:38.774715 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10048> 2013-04-03 10:43:39.593417 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 153 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140438 v187) v2 ==== 289+0+0 (1321037351 0 0) 0x2efa600 con 0x2118420 -10047> 2013-04-03 10:43:39.593491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140438 v187) v2 from mds.? 10.252.58.31:6800/10908 -10046> 2013-04-03 10:43:39.593505 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140438 v187) v2 -- ?+0 0x2a83840 con 0x2118420 -10045> 2013-04-03 10:43:40.939832 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113821 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 153 v187) v2) to leader v1 ==== 560+0+0 (1847686563 0 0) 0x2cc4b00 con 0x208a840 -10044> 2013-04-03 10:43:40.939854 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -10043> 2013-04-03 10:43:40.939865 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc4580 from 10.252.24.88:6789/0 -10042> 2013-04-03 10:43:40.939919 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 153 v187) v2 from mds.0 10.252.24.25:6802/3144 -10041> 2013-04-03 10:43:40.939932 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 153 v187) v2 tid 17678) v2 -- ?+0 0x24f8b40 con 0x208a840 -10040> 2013-04-03 10:43:40.985177 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3003a00 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33111/0 -10039> 2013-04-03 10:43:40.985200 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003a00 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33111/0) -10038> 2013-04-03 10:43:40.985215 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003a00 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -10037> 2013-04-03 10:43:40.985426 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003a00 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -10036> 2013-04-03 10:43:41.391944 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 153 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21809 v187) v2 ==== 289+0+0 (3279871534 0 0) 0x2a848c0 con 0x200d420 -10035> 2013-04-03 10:43:41.391988 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21809 v187) v2 from mds.? 10.252.24.88:6800/17213 -10034> 2013-04-03 10:43:41.391999 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21809 v187) v2 -- ?+0 0x2cc4b00 con 0x200d420 -10033> 2013-04-03 10:43:41.762669 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427461 fc 427429 pn 0 opn 0) v3 -- ?+0 0x300c280 -10032> 2013-04-03 10:43:41.763907 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113822 ==== paxos(lease_ack lc 427461 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (475913018 0 0) 0x300c280 con 0x208a840 -10031> 2013-04-03 10:43:41.763982 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos -10030> 2013-04-03 10:43:41.764003 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394085 -10029> 2013-04-03 10:43:41.764021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -10028> 2013-04-03 10:43:43.262641 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113823 ==== forward(pg_stats(1 pgs tid 28 v 605) v1) to leader v1 ==== 757+0+0 (3962543248 0 0) 0x2efa600 con 0x208a840 -10027> 2013-04-03 10:43:43.262663 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -10026> 2013-04-03 10:43:43.262674 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -10025> 2013-04-03 10:43:43.262725 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 preprocess_query pg_stats(1 pgs tid 28 v 605) v1 from osd.2 10.252.24.30:6801/5656 -10024> 2013-04-03 10:43:43.262750 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 prepare_update pg_stats(1 pgs tid 28 v 605) v1 from osd.2 10.252.24.30:6801/5656 -10023> 2013-04-03 10:43:43.262757 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 prepare_pg_stats pg_stats(1 pgs tid 28 v 605) v1 from osd.2 -10022> 2013-04-03 10:43:43.262764 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -10021> 2013-04-03 10:43:43.312951 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 encode_pending v 29090 -10020> 2013-04-03 10:43:43.312982 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427461) queue_proposal bl 654 bytes; ctx = 0x2c3da70 -10019> 2013-04-03 10:43:43.312992 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427461) propose_queued 427462 654 bytes -10018> 2013-04-03 10:43:43.314536 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427461 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300c000 -10017> 2013-04-03 10:43:43.340217 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113824 ==== paxos(accept lc 427461 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1904195585 0 0) 0x300c000 con 0x208a840 -10016> 2013-04-03 10:43:43.341121 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427462 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300c280 -10015> 2013-04-03 10:43:43.341151 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427462 fc 427429 pn 0 opn 0) v3 -- ?+0 0x300c780 -10014> 2013-04-03 10:43:43.341218 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29089 update_from_paxos applying incremental 29090 -10013> 2013-04-03 10:43:43.341236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 v29090: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s -10012> 2013-04-03 10:43:43.341257 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 send_pg_creates to 0 pgs -10011> 2013-04-03 10:43:43.341261 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 update_logger -10010> 2013-04-03 10:43:43.341271 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 create_pending v 29091 -10009> 2013-04-03 10:43:43.341281 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 _updated_stats for osd.2 10.252.24.30:6801/5656 -10008> 2013-04-03 10:43:43.341288 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 28) v1 tid 17679) v2 -- ?+0 0x24f8d80 con 0x208a840 -10007> 2013-04-03 10:43:43.341297 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 check_osd_map already seen 605 >= 605 -10006> 2013-04-03 10:43:43.341300 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 update_logger -10005> 2013-04-03 10:43:43.341311 7fcdffcdb700 0 log [INF] : pgmap v29090: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s -10004> 2013-04-03 10:43:43.341313 7fcdffcdb700 10 send_log to self -10003> 2013-04-03 10:43:43.341314 7fcdffcdb700 10 log_queue is 1 last_log 156 sent 155 num 1 unsent 1 sending 1 -10002> 2013-04-03 10:43:43.341315 7fcdffcdb700 10 will send 2013-04-03 10:43:43.341312 mon.0 10.252.24.25:6789/0 156 : [INF] pgmap v29090: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s -10001> 2013-04-03 10:43:43.341323 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b78200 -10000> 2013-04-03 10:43:43.341370 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos -9999> 2013-04-03 10:43:43.341390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394085 -9998> 2013-04-03 10:43:43.341612 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9997> 2013-04-03 10:43:43.341635 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b78200 con 0x1bac160 -9996> 2013-04-03 10:43:43.341658 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos -9995> 2013-04-03 10:43:43.341673 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394085 -9994> 2013-04-03 10:43:43.341685 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9993> 2013-04-03 10:43:43.341697 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 preprocess_log log(1 entries) v1 from mon.0 -9992> 2013-04-03 10:43:43.341722 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9991> 2013-04-03 10:43:43.341735 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 prepare_log log(1 entries) v1 from mon.0 -9990> 2013-04-03 10:43:43.341745 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 logging 2013-04-03 10:43:43.341312 mon.0 10.252.24.25:6789/0 156 : [INF] pgmap v29090: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s -9989> 2013-04-03 10:43:43.366369 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113825 ==== paxos(lease_ack lc 427462 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (1849943723 0 0) 0x300c780 con 0x208a840 -9988> 2013-04-03 10:43:43.366407 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos -9987> 2013-04-03 10:43:43.366424 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394085 -9986> 2013-04-03 10:43:43.366441 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9985> 2013-04-03 10:43:43.366790 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 check_down_pgs -9984> 2013-04-03 10:43:43.366821 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 v29090: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s -9983> 2013-04-03 10:43:43.366852 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -9982> 2013-04-03 10:43:43.366874 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -9981> 2013-04-03 10:43:43.366912 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -9980> 2013-04-03 10:43:43.366945 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos -9979> 2013-04-03 10:43:43.366962 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 update_from_paxos version 394085 summary v 394085 -9978> 2013-04-03 10:43:43.366972 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 log -9977> 2013-04-03 10:43:43.366988 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9976> 2013-04-03 10:43:43.367006 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -9975> 2013-04-03 10:43:43.593563 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 154 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140439 v187) v2 ==== 289+0+0 (1842598234 0 0) 0x2efab80 con 0x2118420 -9974> 2013-04-03 10:43:43.593641 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140439 v187) v2 from mds.? 10.252.58.31:6800/10908 -9973> 2013-04-03 10:43:43.593655 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140439 v187) v2 -- ?+0 0x2efa600 con 0x2118420 -9972> 2013-04-03 10:43:44.340421 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394085 encode_pending v394086 -9971> 2013-04-03 10:43:44.340456 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427462) queue_proposal bl 443 bytes; ctx = 0x2c3d7c0 -9970> 2013-04-03 10:43:44.340464 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427462) propose_queued 427463 443 bytes -9969> 2013-04-03 10:43:44.341618 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427462 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003a00 -9968> 2013-04-03 10:43:44.366437 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113826 ==== paxos(accept lc 427462 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1447269130 0 0) 0x3003a00 con 0x208a840 -9967> 2013-04-03 10:43:44.366774 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427463 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300c780 -9966> 2013-04-03 10:43:44.366806 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427463 fc 427429 pn 0 opn 0) v3 -- ?+0 0x300c000 -9965> 2013-04-03 10:43:44.366846 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos -9964> 2013-04-03 10:43:44.366874 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394085 -9963> 2013-04-03 10:43:44.366890 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos latest full 393798 -9962> 2013-04-03 10:43:44.366910 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos applying incremental log 394086 2013-04-03 10:43:43.341312 mon.0 10.252.24.25:6789/0 156 : [INF] pgmap v29090: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s -9961> 2013-04-03 10:43:44.366957 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 check_subs -9960> 2013-04-03 10:43:44.366989 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 create_pending v 394087 -9959> 2013-04-03 10:43:44.367012 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 _updated_log for mon.0 10.252.24.25:6789/0 -9958> 2013-04-03 10:43:44.367026 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 156) v1 -- ?+0 0x22241c0 con 0x1bac160 -9957> 2013-04-03 10:43:44.367078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos -9956> 2013-04-03 10:43:44.367094 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 -9955> 2013-04-03 10:43:44.367112 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9954> 2013-04-03 10:43:44.367138 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 156) v1 ==== 0+0+0 (0 0 0) 0x22241c0 con 0x1bac160 -9953> 2013-04-03 10:43:44.367146 7fcdffcdb700 10 handle_log_ack log(last 156) v1 -9952> 2013-04-03 10:43:44.367147 7fcdffcdb700 10 logged 2013-04-03 10:43:43.341312 mon.0 10.252.24.25:6789/0 156 : [INF] pgmap v29090: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s -9951> 2013-04-03 10:43:44.380148 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113827 ==== paxos(lease_ack lc 427463 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (1041381103 0 0) 0x300c000 con 0x208a840 -9950> 2013-04-03 10:43:44.380189 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos -9949> 2013-04-03 10:43:44.380206 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 -9948> 2013-04-03 10:43:44.380223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9947> 2013-04-03 10:43:44.940015 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113828 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 154 v187) v2) to leader v1 ==== 560+0+0 (2984108091 0 0) 0x2a83dc0 con 0x208a840 -9946> 2013-04-03 10:43:44.940037 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -9945> 2013-04-03 10:43:44.940048 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83000 from 10.252.24.88:6789/0 -9944> 2013-04-03 10:43:44.940101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 154 v187) v2 from mds.0 10.252.24.25:6802/3144 -9943> 2013-04-03 10:43:44.940113 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 154 v187) v2 tid 17680) v2 -- ?+0 0x24f8b40 con 0x208a840 -9942> 2013-04-03 10:43:45.392061 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 154 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21810 v187) v2 ==== 289+0+0 (2798460841 0 0) 0x2a84340 con 0x200d420 -9941> 2013-04-03 10:43:45.392104 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21810 v187) v2 from mds.? 10.252.24.88:6800/17213 -9940> 2013-04-03 10:43:45.392114 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21810 v187) v2 -- ?+0 0x2a83dc0 con 0x200d420 -9939> 2013-04-03 10:43:47.366890 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427463 fc 427429 pn 0 opn 0) v3 -- ?+0 0x303ac80 -9938> 2013-04-03 10:43:47.367743 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113829 ==== paxos(lease_ack lc 427463 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (200377543 0 0) 0x303ac80 con 0x208a840 -9937> 2013-04-03 10:43:47.367816 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos -9936> 2013-04-03 10:43:47.367837 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 -9935> 2013-04-03 10:43:47.367855 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9934> 2013-04-03 10:43:47.419436 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 155 ==== mon_subscribe({mdsmap=188+,monmap=6+}) v2 ==== 42+0+0 (435898880 0 0) 0x2e8b1c0 con 0x200d420 -9933> 2013-04-03 10:43:47.419461 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 -9932> 2013-04-03 10:43:47.419472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 -9931> 2013-04-03 10:43:47.419475 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/17213 -- mon_subscribe_ack(300s) v1 -- ?+0 0x22241c0 -9930> 2013-04-03 10:43:47.593685 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 155 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140440 v187) v2 ==== 289+0+0 (2604884016 0 0) 0x2ef9580 con 0x2118420 -9929> 2013-04-03 10:43:47.593752 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140440 v187) v2 from mds.? 10.252.58.31:6800/10908 -9928> 2013-04-03 10:43:47.593765 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140440 v187) v2 -- ?+0 0x2a84340 con 0x2118420 -9927> 2013-04-03 10:43:48.367146 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 v29090: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2112B/s wr, 0op/s -9926> 2013-04-03 10:43:48.367193 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -9925> 2013-04-03 10:43:48.367216 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -9924> 2013-04-03 10:43:48.367260 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -9923> 2013-04-03 10:43:48.367291 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos -9922> 2013-04-03 10:43:48.367311 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 -9921> 2013-04-03 10:43:48.367321 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 log -9920> 2013-04-03 10:43:48.367338 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9919> 2013-04-03 10:43:48.367357 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -9918> 2013-04-03 10:43:48.940144 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113830 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 155 v187) v2) to leader v1 ==== 560+0+0 (3096723000 0 0) 0x2efab80 con 0x208a840 -9917> 2013-04-03 10:43:48.940166 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -9916> 2013-04-03 10:43:48.940177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc5080 from 10.252.24.88:6789/0 -9915> 2013-04-03 10:43:48.940231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 155 v187) v2 from mds.0 10.252.24.25:6802/3144 -9914> 2013-04-03 10:43:48.940243 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 155 v187) v2 tid 17681) v2 -- ?+0 0x24f8d80 con 0x208a840 -9913> 2013-04-03 10:43:49.392181 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 156 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21811 v187) v2 ==== 289+0+0 (2242093268 0 0) 0x29952c0 con 0x200d420 -9912> 2013-04-03 10:43:49.392254 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21811 v187) v2 from mds.? 10.252.24.88:6800/17213 -9911> 2013-04-03 10:43:49.392267 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21811 v187) v2 -- ?+0 0x2efab80 con 0x200d420 -9910> 2013-04-03 10:43:50.366957 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427463 fc 427429 pn 0 opn 0) v3 -- ?+0 0x303aa00 -9909> 2013-04-03 10:43:50.367763 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113831 ==== paxos(lease_ack lc 427463 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (1658309648 0 0) 0x303aa00 con 0x208a840 -9908> 2013-04-03 10:43:50.367830 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos -9907> 2013-04-03 10:43:50.367851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 -9906> 2013-04-03 10:43:50.367870 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9905> 2013-04-03 10:43:51.593815 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 156 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140441 v187) v2 ==== 289+0+0 (3090014029 0 0) 0x2ef9840 con 0x2118420 -9904> 2013-04-03 10:43:51.593888 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140441 v187) v2 from mds.? 10.252.58.31:6800/10908 -9903> 2013-04-03 10:43:51.593902 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140441 v187) v2 -- ?+0 0x29952c0 con 0x2118420 -9902> 2013-04-03 10:43:51.618200 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113832 ==== forward(pg_stats(1 pgs tid 48 v 605) v1) to leader v1 ==== 753+0+0 (1645573928 0 0) 0x2ef9580 con 0x208a840 -9901> 2013-04-03 10:43:51.618211 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -9900> 2013-04-03 10:43:51.618220 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -9899> 2013-04-03 10:43:51.618254 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 preprocess_query pg_stats(1 pgs tid 48 v 605) v1 from osd.1 10.252.24.25:6805/3572 -9898> 2013-04-03 10:43:51.618279 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 prepare_update pg_stats(1 pgs tid 48 v 605) v1 from osd.1 10.252.24.25:6805/3572 -9897> 2013-04-03 10:43:51.618285 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 prepare_pg_stats pg_stats(1 pgs tid 48 v 605) v1 from osd.1 -9896> 2013-04-03 10:43:51.618292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -9895> 2013-04-03 10:43:51.668480 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 encode_pending v 29091 -9894> 2013-04-03 10:43:51.668519 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427463) queue_proposal bl 650 bytes; ctx = 0x2c3da50 -9893> 2013-04-03 10:43:51.668529 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427463) propose_queued 427464 650 bytes -9892> 2013-04-03 10:43:51.669589 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427463 fc 0 pn 17600 opn 0) v3 -- ?+0 0x303a780 -9891> 2013-04-03 10:43:51.704685 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113833 ==== paxos(accept lc 427463 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3653146804 0 0) 0x303a780 con 0x208a840 -9890> 2013-04-03 10:43:51.705012 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427464 fc 0 pn 17600 opn 0) v3 -- ?+0 0x303aa00 -9889> 2013-04-03 10:43:51.705044 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427464 fc 427429 pn 0 opn 0) v3 -- ?+0 0x303ac80 -9888> 2013-04-03 10:43:51.705115 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29090 update_from_paxos applying incremental 29091 -9887> 2013-04-03 10:43:51.705132 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 v29091: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1165B/s wr, 0op/s -9886> 2013-04-03 10:43:51.705153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 send_pg_creates to 0 pgs -9885> 2013-04-03 10:43:51.705157 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 update_logger -9884> 2013-04-03 10:43:51.705168 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 create_pending v 29092 -9883> 2013-04-03 10:43:51.705177 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 _updated_stats for osd.1 10.252.24.25:6805/3572 -9882> 2013-04-03 10:43:51.705184 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 48) v1 tid 17682) v2 -- ?+0 0x24f8b40 con 0x208a840 -9881> 2013-04-03 10:43:51.705193 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 check_osd_map already seen 605 >= 605 -9880> 2013-04-03 10:43:51.705197 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 update_logger -9879> 2013-04-03 10:43:51.705208 7fcdffcdb700 0 log [INF] : pgmap v29091: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1165B/s wr, 0op/s -9878> 2013-04-03 10:43:51.705209 7fcdffcdb700 10 send_log to self -9877> 2013-04-03 10:43:51.705210 7fcdffcdb700 10 log_queue is 1 last_log 157 sent 156 num 1 unsent 1 sending 1 -9876> 2013-04-03 10:43:51.705212 7fcdffcdb700 10 will send 2013-04-03 10:43:51.705209 mon.0 10.252.24.25:6789/0 157 : [INF] pgmap v29091: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1165B/s wr, 0op/s -9875> 2013-04-03 10:43:51.705218 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b78200 -9874> 2013-04-03 10:43:51.705265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos -9873> 2013-04-03 10:43:51.705284 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 -9872> 2013-04-03 10:43:51.705304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9871> 2013-04-03 10:43:51.705325 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b78200 con 0x1bac160 -9870> 2013-04-03 10:43:51.705349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos -9869> 2013-04-03 10:43:51.705364 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 -9868> 2013-04-03 10:43:51.705374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9867> 2013-04-03 10:43:51.705386 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 preprocess_log log(1 entries) v1 from mon.0 -9866> 2013-04-03 10:43:51.705410 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9865> 2013-04-03 10:43:51.705423 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 prepare_log log(1 entries) v1 from mon.0 -9864> 2013-04-03 10:43:51.705433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 logging 2013-04-03 10:43:51.705209 mon.0 10.252.24.25:6789/0 157 : [INF] pgmap v29091: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1165B/s wr, 0op/s -9863> 2013-04-03 10:43:51.730848 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113834 ==== paxos(lease_ack lc 427464 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3833007919 0 0) 0x303ac80 con 0x208a840 -9862> 2013-04-03 10:43:51.731017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos -9861> 2013-04-03 10:43:51.731038 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 update_from_paxos version 394086 summary v 394086 -9860> 2013-04-03 10:43:51.731054 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9859> 2013-04-03 10:43:51.784046 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113835 ==== forward(pg_stats(1 pgs tid 30456 v 605) v1) to leader v1 ==== 757+0+0 (2769177399 0 0) 0x2cc58c0 con 0x208a840 -9858> 2013-04-03 10:43:51.784059 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -9857> 2013-04-03 10:43:51.784066 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -9856> 2013-04-03 10:43:51.784095 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 preprocess_query pg_stats(1 pgs tid 30456 v 605) v1 from osd.3 10.252.24.88:6801/17538 -9855> 2013-04-03 10:43:51.784116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 prepare_update pg_stats(1 pgs tid 30456 v 605) v1 from osd.3 10.252.24.88:6801/17538 -9854> 2013-04-03 10:43:51.784122 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 prepare_pg_stats pg_stats(1 pgs tid 30456 v 605) v1 from osd.3 -9853> 2013-04-03 10:43:51.784128 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -9852> 2013-04-03 10:43:52.704890 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394086 encode_pending v394087 -9851> 2013-04-03 10:43:52.704923 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427464) queue_proposal bl 443 bytes; ctx = 0x2c3db80 -9850> 2013-04-03 10:43:52.704931 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427464) propose_queued 427465 443 bytes -9849> 2013-04-03 10:43:52.706453 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427464 fc 0 pn 17600 opn 0) v3 -- ?+0 0x303a500 -9848> 2013-04-03 10:43:52.706551 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 encode_pending v 29092 -9847> 2013-04-03 10:43:52.706585 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427429..427464) queue_proposal bl 654 bytes; ctx = 0x2c3db70 -9846> 2013-04-03 10:43:52.706592 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427429..427464) propose_new_value not active; proposal queued -9845> 2013-04-03 10:43:52.730996 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113836 ==== paxos(accept lc 427464 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4245115844 0 0) 0x303a500 con 0x208a840 -9844> 2013-04-03 10:43:52.731325 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427465 fc 0 pn 17600 opn 0) v3 -- ?+0 0x303ac80 -9843> 2013-04-03 10:43:52.731356 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427465 fc 427429 pn 0 opn 0) v3 -- ?+0 0x303a780 -9842> 2013-04-03 10:43:52.731395 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos -9841> 2013-04-03 10:43:52.731422 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos version 394087 summary v 394086 -9840> 2013-04-03 10:43:52.731439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos latest full 393798 -9839> 2013-04-03 10:43:52.731458 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos applying incremental log 394087 2013-04-03 10:43:51.705209 mon.0 10.252.24.25:6789/0 157 : [INF] pgmap v29091: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1165B/s wr, 0op/s -9838> 2013-04-03 10:43:52.731505 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 check_subs -9837> 2013-04-03 10:43:52.731538 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 create_pending v 394088 -9836> 2013-04-03 10:43:52.731562 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 _updated_log for mon.0 10.252.24.25:6789/0 -9835> 2013-04-03 10:43:52.731575 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 157) v1 -- ?+0 0x2224c40 con 0x1bac160 -9834> 2013-04-03 10:43:52.731599 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427465) propose_queued 427466 654 bytes -9833> 2013-04-03 10:43:52.732243 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427465 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300c000 -9832> 2013-04-03 10:43:52.732275 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 157) v1 ==== 0+0+0 (0 0 0) 0x2224c40 con 0x1bac160 -9831> 2013-04-03 10:43:52.732292 7fcdffcdb700 10 handle_log_ack log(last 157) v1 -9830> 2013-04-03 10:43:52.732295 7fcdffcdb700 10 logged 2013-04-03 10:43:51.705209 mon.0 10.252.24.25:6789/0 157 : [INF] pgmap v29091: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1165B/s wr, 0op/s -9829> 2013-04-03 10:43:52.743209 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113837 ==== paxos(lease_ack lc 427465 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (2291983852 0 0) 0x300c000 con 0x208a840 -9828> 2013-04-03 10:43:52.756858 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113838 ==== paxos(accept lc 427465 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3610252929 0 0) 0x303a780 con 0x208a840 -9827> 2013-04-03 10:43:52.758374 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427466 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3051000 -9826> 2013-04-03 10:43:52.758393 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427466 fc 427429 pn 0 opn 0) v3 -- ?+0 0x300c000 -9825> 2013-04-03 10:43:52.758464 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29091 update_from_paxos applying incremental 29092 -9824> 2013-04-03 10:43:52.758485 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 v29092: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -9823> 2013-04-03 10:43:52.758505 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 send_pg_creates to 0 pgs -9822> 2013-04-03 10:43:52.758509 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 update_logger -9821> 2013-04-03 10:43:52.758519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 create_pending v 29093 -9820> 2013-04-03 10:43:52.758918 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 _updated_stats for osd.3 10.252.24.88:6801/17538 -9819> 2013-04-03 10:43:52.758930 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30456) v1 tid 17683) v2 -- ?+0 0x24f8d80 con 0x208a840 -9818> 2013-04-03 10:43:52.758940 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 check_osd_map already seen 605 >= 605 -9817> 2013-04-03 10:43:52.758943 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 update_logger -9816> 2013-04-03 10:43:52.758955 7fcdffcdb700 0 log [INF] : pgmap v29092: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -9815> 2013-04-03 10:43:52.758957 7fcdffcdb700 10 send_log to self -9814> 2013-04-03 10:43:52.758957 7fcdffcdb700 10 log_queue is 1 last_log 158 sent 157 num 1 unsent 1 sending 1 -9813> 2013-04-03 10:43:52.758959 7fcdffcdb700 10 will send 2013-04-03 10:43:52.758956 mon.0 10.252.24.25:6789/0 158 : [INF] pgmap v29092: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -9812> 2013-04-03 10:43:52.758965 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73600 -9811> 2013-04-03 10:43:52.759017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos -9810> 2013-04-03 10:43:52.759035 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos version 394087 summary v 394087 -9809> 2013-04-03 10:43:52.759054 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9808> 2013-04-03 10:43:52.759076 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73600 con 0x1bac160 -9807> 2013-04-03 10:43:52.759098 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos -9806> 2013-04-03 10:43:52.759116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos version 394087 summary v 394087 -9805> 2013-04-03 10:43:52.759125 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9804> 2013-04-03 10:43:52.759138 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 preprocess_log log(1 entries) v1 from mon.0 -9803> 2013-04-03 10:43:52.759159 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9802> 2013-04-03 10:43:52.759172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 prepare_log log(1 entries) v1 from mon.0 -9801> 2013-04-03 10:43:52.759184 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 logging 2013-04-03 10:43:52.758956 mon.0 10.252.24.25:6789/0 158 : [INF] pgmap v29092: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -9800> 2013-04-03 10:43:52.769315 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113839 ==== paxos(lease_ack lc 427466 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (2914900314 0 0) 0x300c000 con 0x208a840 -9799> 2013-04-03 10:43:52.769359 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos -9798> 2013-04-03 10:43:52.769377 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos version 394087 summary v 394087 -9797> 2013-04-03 10:43:52.769394 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9796> 2013-04-03 10:43:52.940214 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113840 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 156 v187) v2) to leader v1 ==== 560+0+0 (507622997 0 0) 0x29968c0 con 0x208a840 -9795> 2013-04-03 10:43:52.940227 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -9794> 2013-04-03 10:43:52.940236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2996600 from 10.252.24.88:6789/0 -9793> 2013-04-03 10:43:52.940268 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 156 v187) v2 from mds.0 10.252.24.25:6802/3144 -9792> 2013-04-03 10:43:52.940279 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 156 v187) v2 tid 17684) v2 -- ?+0 0x24f8b40 con 0x208a840 -9791> 2013-04-03 10:43:53.332508 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_start_round curr 4 -9790> 2013-04-03 10:43:53.332531 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_start_round new 5 -9789> 2013-04-03 10:43:53.332538 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck -9788> 2013-04-03 10:43:53.332542 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck start timecheck epoch 352 round 5 -9787> 2013-04-03 10:43:53.332553 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck send time_check( ping e 352 r 5 ) v1 to mon.2 10.252.24.88:6789/0 -9786> 2013-04-03 10:43:53.332567 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- time_check( ping e 352 r 5 ) v1 -- ?+0 0x24f8000 -9785> 2013-04-03 10:43:53.332582 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_start_round setting up next event -9784> 2013-04-03 10:43:53.333308 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113841 ==== time_check( pong e 352 r 5 ts 2013-04-03 10:43:53.332381 ) v1 ==== 36+0+0 (3995498854 0 0) 0x24f8000 con 0x208a840 -9783> 2013-04-03 10:43:53.333327 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck time_check( pong e 352 r 5 ts 2013-04-03 10:43:53.332381 ) v1 -9782> 2013-04-03 10:43:53.333333 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck_leader time_check( pong e 352 r 5 ts 2013-04-03 10:43:53.332381 ) v1 -9781> 2013-04-03 10:43:53.333340 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck_leader from mon.2 10.252.24.88:6789/0 ts 2013-04-03 10:43:53.332381 delta -0.000956059 skew_bound -0.000170059 latency 0.000786 -9780> 2013-04-03 10:43:53.333357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_timecheck_leader got pongs from everybody (2 total) -9779> 2013-04-03 10:43:53.333360 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_finish_round curr 5 -9778> 2013-04-03 10:43:53.333361 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_report -9777> 2013-04-03 10:43:53.333366 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 timecheck_report send report to mon.2 10.252.24.88:6789/0 -9776> 2013-04-03 10:43:53.333369 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- time_check( report e 352 r 6 #skews 2 #latencies 2 ) v1 -- ?+0 0x24f8d80 -9775> 2013-04-03 10:43:53.367490 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 check_down_pgs -9774> 2013-04-03 10:43:53.367520 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 v29092: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -9773> 2013-04-03 10:43:53.367550 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -9772> 2013-04-03 10:43:53.367574 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -9771> 2013-04-03 10:43:53.367611 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -9770> 2013-04-03 10:43:53.367642 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos -9769> 2013-04-03 10:43:53.367663 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 update_from_paxos version 394087 summary v 394087 -9768> 2013-04-03 10:43:53.367673 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 log -9767> 2013-04-03 10:43:53.367693 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9766> 2013-04-03 10:43:53.367713 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -9765> 2013-04-03 10:43:53.392255 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 157 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21812 v187) v2 ==== 289+0+0 (1839897767 0 0) 0x2996b80 con 0x200d420 -9764> 2013-04-03 10:43:53.392315 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21812 v187) v2 from mds.? 10.252.24.88:6800/17213 -9763> 2013-04-03 10:43:53.392327 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21812 v187) v2 -- ?+0 0x29968c0 con 0x200d420 -9762> 2013-04-03 10:43:53.757058 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394087 encode_pending v394088 -9761> 2013-04-03 10:43:53.757098 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427466) queue_proposal bl 442 bytes; ctx = 0x2c3da60 -9760> 2013-04-03 10:43:53.757107 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427466) propose_queued 427467 442 bytes -9759> 2013-04-03 10:43:53.758216 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427466 fc 0 pn 17600 opn 0) v3 -- ?+0 0x303a280 -9758> 2013-04-03 10:43:53.782196 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113842 ==== paxos(accept lc 427466 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1236359550 0 0) 0x303a280 con 0x208a840 -9757> 2013-04-03 10:43:53.782540 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427467 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300c000 -9756> 2013-04-03 10:43:53.782570 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427467 fc 427429 pn 0 opn 0) v3 -- ?+0 0x303a780 -9755> 2013-04-03 10:43:53.782611 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos -9754> 2013-04-03 10:43:53.782639 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394087 -9753> 2013-04-03 10:43:53.782656 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos latest full 393798 -9752> 2013-04-03 10:43:53.782675 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos applying incremental log 394088 2013-04-03 10:43:52.758956 mon.0 10.252.24.25:6789/0 158 : [INF] pgmap v29092: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -9751> 2013-04-03 10:43:53.782722 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 check_subs -9750> 2013-04-03 10:43:53.782755 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 create_pending v 394089 -9749> 2013-04-03 10:43:53.782778 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 _updated_log for mon.0 10.252.24.25:6789/0 -9748> 2013-04-03 10:43:53.782792 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 158) v1 -- ?+0 0x2224c40 con 0x1bac160 -9747> 2013-04-03 10:43:53.782847 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos -9746> 2013-04-03 10:43:53.782864 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394088 -9745> 2013-04-03 10:43:53.782882 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9744> 2013-04-03 10:43:53.782906 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 158) v1 ==== 0+0+0 (0 0 0) 0x2224c40 con 0x1bac160 -9743> 2013-04-03 10:43:53.782914 7fcdffcdb700 10 handle_log_ack log(last 158) v1 -9742> 2013-04-03 10:43:53.782915 7fcdffcdb700 10 logged 2013-04-03 10:43:52.758956 mon.0 10.252.24.25:6789/0 158 : [INF] pgmap v29092: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 545B/s wr, 0op/s -9741> 2013-04-03 10:43:53.794444 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113843 ==== paxos(lease_ack lc 427467 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (1015080834 0 0) 0x303a780 con 0x208a840 -9740> 2013-04-03 10:43:53.794484 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos -9739> 2013-04-03 10:43:53.794501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394088 -9738> 2013-04-03 10:43:53.794517 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9737> 2013-04-03 10:43:55.484943 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 157 ==== mon_subscribe({mdsmap=188+,monmap=6+}) v2 ==== 42+0+0 (435898880 0 0) 0x2e8b700 con 0x2118420 -9736> 2013-04-03 10:43:55.484970 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 -9735> 2013-04-03 10:43:55.484981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 -9734> 2013-04-03 10:43:55.484984 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.58.31:6800/10908 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2224c40 -9733> 2013-04-03 10:43:55.593892 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 158 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140442 v187) v2 ==== 289+0+0 (3718127306 0 0) 0x2efa080 con 0x2118420 -9732> 2013-04-03 10:43:55.593965 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140442 v187) v2 from mds.? 10.252.58.31:6800/10908 -9731> 2013-04-03 10:43:55.593978 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140442 v187) v2 -- ?+0 0x2996b80 con 0x2118420 -9730> 2013-04-03 10:43:56.782649 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427467 fc 427429 pn 0 opn 0) v3 -- ?+0 0x303a000 -9729> 2013-04-03 10:43:56.783379 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113844 ==== paxos(lease_ack lc 427467 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (215798387 0 0) 0x303a000 con 0x208a840 -9728> 2013-04-03 10:43:56.783447 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos -9727> 2013-04-03 10:43:56.783471 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394088 -9726> 2013-04-03 10:43:56.783489 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9725> 2013-04-03 10:43:56.940325 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113845 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 157 v187) v2) to leader v1 ==== 560+0+0 (386426966 0 0) 0x2cc58c0 con 0x208a840 -9724> 2013-04-03 10:43:56.940338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -9723> 2013-04-03 10:43:56.940347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2995b00 from 10.252.24.88:6789/0 -9722> 2013-04-03 10:43:56.940380 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 157 v187) v2 from mds.0 10.252.24.25:6802/3144 -9721> 2013-04-03 10:43:56.940392 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 157 v187) v2 tid 17685) v2 -- ?+0 0x24f8000 con 0x208a840 -9720> 2013-04-03 10:43:57.392373 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 158 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21813 v187) v2 ==== 289+0+0 (1321477082 0 0) 0x2996080 con 0x200d420 -9719> 2013-04-03 10:43:57.392445 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21813 v187) v2 from mds.? 10.252.24.88:6800/17213 -9718> 2013-04-03 10:43:57.392459 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21813 v187) v2 -- ?+0 0x2996340 con 0x200d420 -9717> 2013-04-03 10:43:58.264517 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113846 ==== forward(pg_stats(1 pgs tid 29 v 605) v1) to leader v1 ==== 757+0+0 (2223231494 0 0) 0x2efa080 con 0x208a840 -9716> 2013-04-03 10:43:58.264545 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -9715> 2013-04-03 10:43:58.264556 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 -9714> 2013-04-03 10:43:58.264607 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 preprocess_query pg_stats(1 pgs tid 29 v 605) v1 from osd.2 10.252.24.30:6801/5656 -9713> 2013-04-03 10:43:58.264632 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 prepare_update pg_stats(1 pgs tid 29 v 605) v1 from osd.2 10.252.24.30:6801/5656 -9712> 2013-04-03 10:43:58.264639 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 prepare_pg_stats pg_stats(1 pgs tid 29 v 605) v1 from osd.2 -9711> 2013-04-03 10:43:58.264645 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -9710> 2013-04-03 10:43:58.314833 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 encode_pending v 29093 -9709> 2013-04-03 10:43:58.314870 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427467) queue_proposal bl 654 bytes; ctx = 0x2c3d7a0 -9708> 2013-04-03 10:43:58.314880 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427467) propose_queued 427468 654 bytes -9707> 2013-04-03 10:43:58.316159 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427467 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062c80 -9706> 2013-04-03 10:43:58.341128 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113847 ==== paxos(accept lc 427467 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1095035855 0 0) 0x3062c80 con 0x208a840 -9705> 2013-04-03 10:43:58.341418 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427468 fc 0 pn 17600 opn 0) v3 -- ?+0 0x303a000 -9704> 2013-04-03 10:43:58.341450 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427468 fc 427429 pn 0 opn 0) v3 -- ?+0 0x303a780 -9703> 2013-04-03 10:43:58.341521 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29092 update_from_paxos applying incremental 29093 -9702> 2013-04-03 10:43:58.341536 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 v29093: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2619B/s wr, 0op/s -9701> 2013-04-03 10:43:58.341557 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 send_pg_creates to 0 pgs -9700> 2013-04-03 10:43:58.341561 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 update_logger -9699> 2013-04-03 10:43:58.341572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 create_pending v 29094 -9698> 2013-04-03 10:43:58.341581 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 _updated_stats for osd.2 10.252.24.30:6801/5656 -9697> 2013-04-03 10:43:58.341588 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 29) v1 tid 17686) v2 -- ?+0 0x24f8d80 con 0x208a840 -9696> 2013-04-03 10:43:58.341598 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 check_osd_map already seen 605 >= 605 -9695> 2013-04-03 10:43:58.341600 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 update_logger -9694> 2013-04-03 10:43:58.341611 7fcdffcdb700 0 log [INF] : pgmap v29093: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2619B/s wr, 0op/s -9693> 2013-04-03 10:43:58.341612 7fcdffcdb700 10 send_log to self -9692> 2013-04-03 10:43:58.341613 7fcdffcdb700 10 log_queue is 1 last_log 159 sent 158 num 1 unsent 1 sending 1 -9691> 2013-04-03 10:43:58.341615 7fcdffcdb700 10 will send 2013-04-03 10:43:58.341612 mon.0 10.252.24.25:6789/0 159 : [INF] pgmap v29093: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2619B/s wr, 0op/s -9690> 2013-04-03 10:43:58.341621 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73600 -9689> 2013-04-03 10:43:58.341668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos -9688> 2013-04-03 10:43:58.341688 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394088 -9687> 2013-04-03 10:43:58.341707 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9686> 2013-04-03 10:43:58.341729 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73600 con 0x1bac160 -9685> 2013-04-03 10:43:58.341754 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos -9684> 2013-04-03 10:43:58.341770 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394088 -9683> 2013-04-03 10:43:58.341779 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9682> 2013-04-03 10:43:58.341791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 preprocess_log log(1 entries) v1 from mon.0 -9681> 2013-04-03 10:43:58.341827 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9680> 2013-04-03 10:43:58.341843 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 prepare_log log(1 entries) v1 from mon.0 -9679> 2013-04-03 10:43:58.341855 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 logging 2013-04-03 10:43:58.341612 mon.0 10.252.24.25:6789/0 159 : [INF] pgmap v29093: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2619B/s wr, 0op/s -9678> 2013-04-03 10:43:58.354826 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113848 ==== paxos(lease_ack lc 427468 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (4099437758 0 0) 0x303a780 con 0x208a840 -9677> 2013-04-03 10:43:58.354865 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos -9676> 2013-04-03 10:43:58.354882 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394088 -9675> 2013-04-03 10:43:58.354897 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9674> 2013-04-03 10:43:58.367825 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 check_down_pgs -9673> 2013-04-03 10:43:58.367855 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29093 v29093: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2619B/s wr, 0op/s -9672> 2013-04-03 10:43:58.367881 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -9671> 2013-04-03 10:43:58.367903 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -9670> 2013-04-03 10:43:58.367941 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -9669> 2013-04-03 10:43:58.367972 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos -9668> 2013-04-03 10:43:58.367991 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 update_from_paxos version 394088 summary v 394088 -9667> 2013-04-03 10:43:58.368000 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394088 log -9666> 2013-04-03 10:43:58.368017 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9665> 2013-04-03 10:43:58.368034 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -9664> 2013-04-03 10:43:58.759465 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) -9663> 2013-04-03 10:43:58.759504 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader -9662> 2013-04-03 10:43:59.341346 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394088 encode_pending v394089 -9661> 2013-04-03 10:43:59.341376 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427468) queue_proposal bl 443 bytes; ctx = 0x2c3da20 -9660> 2013-04-03 10:43:59.341384 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427468) propose_queued 427469 443 bytes -9659> 2013-04-03 10:43:59.342535 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427468 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062a00 -9658> 2013-04-03 10:43:59.372646 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113849 ==== paxos(accept lc 427468 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (517133862 0 0) 0x3062a00 con 0x208a840 -9657> 2013-04-03 10:43:59.372954 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427469 fc 0 pn 17600 opn 0) v3 -- ?+0 0x303a780 -9656> 2013-04-03 10:43:59.372986 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427469 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3062c80 -9655> 2013-04-03 10:43:59.373035 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos -9654> 2013-04-03 10:43:59.373062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394088 -9653> 2013-04-03 10:43:59.373076 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos latest full 393798 -9652> 2013-04-03 10:43:59.373095 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394089 update_from_paxos applying incremental log 394089 2013-04-03 10:43:58.341612 mon.0 10.252.24.25:6789/0 159 : [INF] pgmap v29093: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2619B/s wr, 0op/s -9651> 2013-04-03 10:43:59.373355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 check_subs -9650> 2013-04-03 10:43:59.373388 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 create_pending v 394090 -9649> 2013-04-03 10:43:59.373410 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394089 _updated_log for mon.0 10.252.24.25:6789/0 -9648> 2013-04-03 10:43:59.373423 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 159) v1 -- ?+0 0x2224000 con 0x1bac160 -9647> 2013-04-03 10:43:59.373475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos -9646> 2013-04-03 10:43:59.373491 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 -9645> 2013-04-03 10:43:59.373509 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9644> 2013-04-03 10:43:59.373531 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 159) v1 ==== 0+0+0 (0 0 0) 0x2224000 con 0x1bac160 -9643> 2013-04-03 10:43:59.373539 7fcdffcdb700 10 handle_log_ack log(last 159) v1 -9642> 2013-04-03 10:43:59.373542 7fcdffcdb700 10 logged 2013-04-03 10:43:58.341612 mon.0 10.252.24.25:6789/0 159 : [INF] pgmap v29093: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2619B/s wr, 0op/s -9641> 2013-04-03 10:43:59.384996 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113850 ==== paxos(lease_ack lc 427469 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (469366406 0 0) 0x3062c80 con 0x208a840 -9640> 2013-04-03 10:43:59.385031 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos -9639> 2013-04-03 10:43:59.385047 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 -9638> 2013-04-03 10:43:59.385080 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9637> 2013-04-03 10:43:59.594041 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 159 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140443 v187) v2 ==== 289+0+0 (4277182903 0 0) 0x2995580 con 0x2118420 -9636> 2013-04-03 10:43:59.594119 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140443 v187) v2 from mds.? 10.252.58.31:6800/10908 -9635> 2013-04-03 10:43:59.594133 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140443 v187) v2 -- ?+0 0x2efa080 con 0x2118420 -9634> 2013-04-03 10:44:00.940476 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113851 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 158 v187) v2) to leader v1 ==== 560+0+0 (1086735201 0 0) 0x2a51dc0 con 0x208a840 -9633> 2013-04-03 10:44:00.940497 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -9632> 2013-04-03 10:44:00.940506 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2995000 from 10.252.24.88:6789/0 -9631> 2013-04-03 10:44:00.940561 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 158 v187) v2 from mds.0 10.252.24.25:6802/3144 -9630> 2013-04-03 10:44:00.940573 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 158 v187) v2 tid 17687) v2 -- ?+0 0x24f8000 con 0x208a840 -9629> 2013-04-03 10:44:01.017183 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3003780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33115/0 -9628> 2013-04-03 10:44:01.017205 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33115/0) -9627> 2013-04-03 10:44:01.017223 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003780 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -9626> 2013-04-03 10:44:01.017420 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003780 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -9625> 2013-04-03 10:44:01.392454 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 159 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21814 v187) v2 ==== 289+0+0 (729277021 0 0) 0x2f7a600 con 0x200d420 -9624> 2013-04-03 10:44:01.392526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21814 v187) v2 from mds.? 10.252.24.88:6800/17213 -9623> 2013-04-03 10:44:01.392538 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21814 v187) v2 -- ?+0 0x2a51dc0 con 0x200d420 -9622> 2013-04-03 10:44:02.373063 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427469 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3062780 -9621> 2013-04-03 10:44:02.373856 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113852 ==== paxos(lease_ack lc 427469 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (1718688735 0 0) 0x3062780 con 0x208a840 -9620> 2013-04-03 10:44:02.373924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos -9619> 2013-04-03 10:44:02.373943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 -9618> 2013-04-03 10:44:02.373960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9617> 2013-04-03 10:44:03.368175 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29093 v29093: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 2619B/s wr, 0op/s -9616> 2013-04-03 10:44:03.368221 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -9615> 2013-04-03 10:44:03.368243 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -9614> 2013-04-03 10:44:03.368283 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -9613> 2013-04-03 10:44:03.368315 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos -9612> 2013-04-03 10:44:03.368333 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 -9611> 2013-04-03 10:44:03.368343 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394089 log -9610> 2013-04-03 10:44:03.368360 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9609> 2013-04-03 10:44:03.368381 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -9608> 2013-04-03 10:44:03.594175 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 160 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140444 v187) v2 ==== 289+0+0 (385457604 0 0) 0x2cc42c0 con 0x2118420 -9607> 2013-04-03 10:44:03.594255 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140444 v187) v2 from mds.? 10.252.58.31:6800/10908 -9606> 2013-04-03 10:44:03.594269 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140444 v187) v2 -- ?+0 0x2f7a600 con 0x2118420 -9605> 2013-04-03 10:44:04.940536 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113853 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 159 v187) v2) to leader v1 ==== 560+0+0 (2405497165 0 0) 0x2995580 con 0x208a840 -9604> 2013-04-03 10:44:04.940557 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -9603> 2013-04-03 10:44:04.940566 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a84600 from 10.252.24.88:6789/0 -9602> 2013-04-03 10:44:04.940621 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 159 v187) v2 from mds.0 10.252.24.25:6802/3144 -9601> 2013-04-03 10:44:04.940631 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 159 v187) v2 tid 17688) v2 -- ?+0 0x24f8d80 con 0x208a840 -9600> 2013-04-03 10:44:05.373133 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427469 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3062500 -9599> 2013-04-03 10:44:05.373931 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113854 ==== paxos(lease_ack lc 427469 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (2105094209 0 0) 0x3062500 con 0x208a840 -9598> 2013-04-03 10:44:05.373998 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos -9597> 2013-04-03 10:44:05.374020 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 -9596> 2013-04-03 10:44:05.374038 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9595> 2013-04-03 10:44:05.392567 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 160 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21815 v187) v2 ==== 289+0+0 (135879968 0 0) 0x2145000 con 0x200d420 -9594> 2013-04-03 10:44:05.392618 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21815 v187) v2 from mds.? 10.252.24.88:6800/17213 -9593> 2013-04-03 10:44:05.392629 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21815 v187) v2 -- ?+0 0x2995580 con 0x200d420 -9592> 2013-04-03 10:44:06.619963 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113855 ==== forward(pg_stats(1 pgs tid 49 v 605) v1) to leader v1 ==== 753+0+0 (3134434981 0 0) 0x2cc42c0 con 0x208a840 -9591> 2013-04-03 10:44:06.619984 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -9590> 2013-04-03 10:44:06.619994 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -9589> 2013-04-03 10:44:06.620045 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29093 preprocess_query pg_stats(1 pgs tid 49 v 605) v1 from osd.1 10.252.24.25:6805/3572 -9588> 2013-04-03 10:44:06.620069 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29093 prepare_update pg_stats(1 pgs tid 49 v 605) v1 from osd.1 10.252.24.25:6805/3572 -9587> 2013-04-03 10:44:06.620075 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29093 prepare_pg_stats pg_stats(1 pgs tid 49 v 605) v1 from osd.1 -9586> 2013-04-03 10:44:06.620081 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29093 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -9585> 2013-04-03 10:44:06.670268 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29093 encode_pending v 29094 -9584> 2013-04-03 10:44:06.670301 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427469) queue_proposal bl 650 bytes; ctx = 0x2c3da10 -9583> 2013-04-03 10:44:06.670309 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427469) propose_queued 427470 650 bytes -9582> 2013-04-03 10:44:06.671468 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427469 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062280 -9581> 2013-04-03 10:44:06.807381 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113856 ==== paxos(accept lc 427469 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1299485555 0 0) 0x3062280 con 0x208a840 -9580> 2013-04-03 10:44:06.807725 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427470 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062500 -9579> 2013-04-03 10:44:06.807755 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427470 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3062780 -9578> 2013-04-03 10:44:06.807832 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29093 update_from_paxos applying incremental 29094 -9577> 2013-04-03 10:44:06.807848 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 v29094: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1466B/s wr, 0op/s -9576> 2013-04-03 10:44:06.807866 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 send_pg_creates to 0 pgs -9575> 2013-04-03 10:44:06.807870 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 update_logger -9574> 2013-04-03 10:44:06.807882 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 create_pending v 29095 -9573> 2013-04-03 10:44:06.807891 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29094 _updated_stats for osd.1 10.252.24.25:6805/3572 -9572> 2013-04-03 10:44:06.807898 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 49) v1 tid 17689) v2 -- ?+0 0x24f8000 con 0x208a840 -9571> 2013-04-03 10:44:06.807907 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 check_osd_map already seen 605 >= 605 -9570> 2013-04-03 10:44:06.807910 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 update_logger -9569> 2013-04-03 10:44:06.807920 7fcdffcdb700 0 log [INF] : pgmap v29094: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1466B/s wr, 0op/s -9568> 2013-04-03 10:44:06.807922 7fcdffcdb700 10 send_log to self -9567> 2013-04-03 10:44:06.807923 7fcdffcdb700 10 log_queue is 1 last_log 160 sent 159 num 1 unsent 1 sending 1 -9566> 2013-04-03 10:44:06.807924 7fcdffcdb700 10 will send 2013-04-03 10:44:06.807921 mon.0 10.252.24.25:6789/0 160 : [INF] pgmap v29094: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1466B/s wr, 0op/s -9565> 2013-04-03 10:44:06.807930 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73600 -9564> 2013-04-03 10:44:06.807978 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos -9563> 2013-04-03 10:44:06.807997 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 -9562> 2013-04-03 10:44:06.808016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9561> 2013-04-03 10:44:06.808037 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73600 con 0x1bac160 -9560> 2013-04-03 10:44:06.808058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos -9559> 2013-04-03 10:44:06.808073 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 -9558> 2013-04-03 10:44:06.808082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9557> 2013-04-03 10:44:06.808092 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 preprocess_log log(1 entries) v1 from mon.0 -9556> 2013-04-03 10:44:06.808115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9555> 2013-04-03 10:44:06.808130 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 prepare_log log(1 entries) v1 from mon.0 -9554> 2013-04-03 10:44:06.808140 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 logging 2013-04-03 10:44:06.807921 mon.0 10.252.24.25:6789/0 160 : [INF] pgmap v29094: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1466B/s wr, 0op/s -9553> 2013-04-03 10:44:06.819780 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113857 ==== paxos(lease_ack lc 427470 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3215293164 0 0) 0x3062780 con 0x208a840 -9552> 2013-04-03 10:44:06.819822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos -9551> 2013-04-03 10:44:06.819839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394089 update_from_paxos version 394089 summary v 394089 -9550> 2013-04-03 10:44:06.819853 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9549> 2013-04-03 10:44:07.594278 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 161 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140445 v187) v2 ==== 289+0+0 (899097273 0 0) 0x2f7a340 con 0x2118420 -9548> 2013-04-03 10:44:07.594353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140445 v187) v2 from mds.? 10.252.58.31:6800/10908 -9547> 2013-04-03 10:44:07.594366 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140445 v187) v2 -- ?+0 0x2cc42c0 con 0x2118420 -9546> 2013-04-03 10:44:07.807566 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394089 encode_pending v394090 -9545> 2013-04-03 10:44:07.807592 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427470) queue_proposal bl 443 bytes; ctx = 0x2c3db50 -9544> 2013-04-03 10:44:07.807600 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427470) propose_queued 427471 443 bytes -9543> 2013-04-03 10:44:07.809075 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427470 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062000 -9542> 2013-04-03 10:44:07.837615 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113858 ==== paxos(accept lc 427470 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (368152880 0 0) 0x3062000 con 0x208a840 -9541> 2013-04-03 10:44:07.838521 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427471 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062780 -9540> 2013-04-03 10:44:07.838551 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427471 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3062280 -9539> 2013-04-03 10:44:07.838590 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos -9538> 2013-04-03 10:44:07.838617 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394089 -9537> 2013-04-03 10:44:07.838632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos latest full 393798 -9536> 2013-04-03 10:44:07.838652 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394090 update_from_paxos applying incremental log 394090 2013-04-03 10:44:06.807921 mon.0 10.252.24.25:6789/0 160 : [INF] pgmap v29094: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1466B/s wr, 0op/s -9535> 2013-04-03 10:44:07.838699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 check_subs -9534> 2013-04-03 10:44:07.838731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 create_pending v 394091 -9533> 2013-04-03 10:44:07.838754 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394090 _updated_log for mon.0 10.252.24.25:6789/0 -9532> 2013-04-03 10:44:07.838769 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 160) v1 -- ?+0 0x2e8b380 con 0x1bac160 -9531> 2013-04-03 10:44:07.838819 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos -9530> 2013-04-03 10:44:07.838835 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 -9529> 2013-04-03 10:44:07.838853 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9528> 2013-04-03 10:44:07.838874 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 160) v1 ==== 0+0+0 (0 0 0) 0x2e8b380 con 0x1bac160 -9527> 2013-04-03 10:44:07.838882 7fcdffcdb700 10 handle_log_ack log(last 160) v1 -9526> 2013-04-03 10:44:07.838883 7fcdffcdb700 10 logged 2013-04-03 10:44:06.807921 mon.0 10.252.24.25:6789/0 160 : [INF] pgmap v29094: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1466B/s wr, 0op/s -9525> 2013-04-03 10:44:07.863763 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113859 ==== paxos(lease_ack lc 427471 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (960770853 0 0) 0x3062280 con 0x208a840 -9524> 2013-04-03 10:44:07.863804 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos -9523> 2013-04-03 10:44:07.863820 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 -9522> 2013-04-03 10:44:07.863837 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9521> 2013-04-03 10:44:08.368529 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29094 check_down_pgs -9520> 2013-04-03 10:44:08.368556 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29094 v29094: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1466B/s wr, 0op/s -9519> 2013-04-03 10:44:08.368584 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -9518> 2013-04-03 10:44:08.368611 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -9517> 2013-04-03 10:44:08.368647 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -9516> 2013-04-03 10:44:08.368680 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos -9515> 2013-04-03 10:44:08.368698 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 -9514> 2013-04-03 10:44:08.368707 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394090 log -9513> 2013-04-03 10:44:08.368724 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9512> 2013-04-03 10:44:08.368743 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -9511> 2013-04-03 10:44:08.940822 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113860 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 160 v187) v2) to leader v1 ==== 560+0+0 (800675161 0 0) 0x2995840 con 0x208a840 -9510> 2013-04-03 10:44:08.940844 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -9509> 2013-04-03 10:44:08.940854 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a51840 from 10.252.24.88:6789/0 -9508> 2013-04-03 10:44:08.940908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 160 v187) v2 from mds.0 10.252.24.25:6802/3144 -9507> 2013-04-03 10:44:08.940919 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 160 v187) v2 tid 17690) v2 -- ?+0 0x24f8d80 con 0x208a840 -9506> 2013-04-03 10:44:09.392749 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 161 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21816 v187) v2 ==== 289+0+0 (4270451786 0 0) 0x2a52600 con 0x200d420 -9505> 2013-04-03 10:44:09.392827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21816 v187) v2 from mds.? 10.252.24.88:6800/17213 -9504> 2013-04-03 10:44:09.392840 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21816 v187) v2 -- ?+0 0x2995840 con 0x200d420 -9503> 2013-04-03 10:44:10.838630 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427471 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3003780 -9502> 2013-04-03 10:44:10.839363 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113861 ==== paxos(lease_ack lc 427471 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (538460779 0 0) 0x3003780 con 0x208a840 -9501> 2013-04-03 10:44:10.839432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos -9500> 2013-04-03 10:44:10.839452 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 -9499> 2013-04-03 10:44:10.839469 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9498> 2013-04-03 10:44:11.594430 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 162 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140446 v187) v2 ==== 289+0+0 (1344594750 0 0) 0x2f7a8c0 con 0x2118420 -9497> 2013-04-03 10:44:11.594507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140446 v187) v2 from mds.? 10.252.58.31:6800/10908 -9496> 2013-04-03 10:44:11.594521 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140446 v187) v2 -- ?+0 0x2a52600 con 0x2118420 -9495> 2013-04-03 10:44:12.940870 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113862 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 161 v187) v2) to leader v1 ==== 560+0+0 (653298522 0 0) 0x2f7a340 con 0x208a840 -9494> 2013-04-03 10:44:12.940890 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -9493> 2013-04-03 10:44:12.940899 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2995dc0 from 10.252.24.88:6789/0 -9492> 2013-04-03 10:44:12.940952 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 161 v187) v2 from mds.0 10.252.24.25:6802/3144 -9491> 2013-04-03 10:44:12.940962 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 161 v187) v2 tid 17691) v2 -- ?+0 0x24f8000 con 0x208a840 -9490> 2013-04-03 10:44:13.266530 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113863 ==== forward(pg_stats(1 pgs tid 30 v 605) v1) to leader v1 ==== 757+0+0 (976383372 0 0) 0x2f7a8c0 con 0x208a840 -9489> 2013-04-03 10:44:13.266559 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -9488> 2013-04-03 10:44:13.266569 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 -9487> 2013-04-03 10:44:13.266620 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 preprocess_query pg_stats(1 pgs tid 30 v 605) v1 from osd.2 10.252.24.30:6801/5656 -9486> 2013-04-03 10:44:13.266645 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 prepare_update pg_stats(1 pgs tid 30 v 605) v1 from osd.2 10.252.24.30:6801/5656 -9485> 2013-04-03 10:44:13.266650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 prepare_pg_stats pg_stats(1 pgs tid 30 v 605) v1 from osd.2 -9484> 2013-04-03 10:44:13.266657 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29094 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -9483> 2013-04-03 10:44:13.316847 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29094 encode_pending v 29095 -9482> 2013-04-03 10:44:13.316888 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427471) queue_proposal bl 654 bytes; ctx = 0x2c3da00 -9481> 2013-04-03 10:44:13.316897 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427471) propose_queued 427472 654 bytes -9480> 2013-04-03 10:44:13.317862 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427471 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3083c80 -9479> 2013-04-03 10:44:13.344947 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113864 ==== paxos(accept lc 427471 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (68913505 0 0) 0x3083c80 con 0x208a840 -9478> 2013-04-03 10:44:13.345303 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427472 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003780 -9477> 2013-04-03 10:44:13.345330 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427472 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3062280 -9476> 2013-04-03 10:44:13.345403 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29094 update_from_paxos applying incremental 29095 -9475> 2013-04-03 10:44:13.345418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 v29095: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -9474> 2013-04-03 10:44:13.345437 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 send_pg_creates to 0 pgs -9473> 2013-04-03 10:44:13.345440 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 update_logger -9472> 2013-04-03 10:44:13.345449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 create_pending v 29096 -9471> 2013-04-03 10:44:13.345458 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29095 _updated_stats for osd.2 10.252.24.30:6801/5656 -9470> 2013-04-03 10:44:13.345464 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30) v1 tid 17692) v2 -- ?+0 0x24f8d80 con 0x208a840 -9469> 2013-04-03 10:44:13.345477 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 check_osd_map already seen 605 >= 605 -9468> 2013-04-03 10:44:13.345480 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 update_logger -9467> 2013-04-03 10:44:13.345491 7fcdffcdb700 0 log [INF] : pgmap v29095: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -9466> 2013-04-03 10:44:13.345493 7fcdffcdb700 10 send_log to self -9465> 2013-04-03 10:44:13.345493 7fcdffcdb700 10 log_queue is 1 last_log 161 sent 160 num 1 unsent 1 sending 1 -9464> 2013-04-03 10:44:13.345495 7fcdffcdb700 10 will send 2013-04-03 10:44:13.345492 mon.0 10.252.24.25:6789/0 161 : [INF] pgmap v29095: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -9463> 2013-04-03 10:44:13.345502 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73600 -9462> 2013-04-03 10:44:13.345686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos -9461> 2013-04-03 10:44:13.345709 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 -9460> 2013-04-03 10:44:13.345729 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9459> 2013-04-03 10:44:13.345752 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73600 con 0x1bac160 -9458> 2013-04-03 10:44:13.345775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos -9457> 2013-04-03 10:44:13.345790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 -9456> 2013-04-03 10:44:13.345798 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9455> 2013-04-03 10:44:13.345808 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 preprocess_log log(1 entries) v1 from mon.0 -9454> 2013-04-03 10:44:13.345843 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9453> 2013-04-03 10:44:13.345857 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 prepare_log log(1 entries) v1 from mon.0 -9452> 2013-04-03 10:44:13.345871 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 logging 2013-04-03 10:44:13.345492 mon.0 10.252.24.25:6789/0 161 : [INF] pgmap v29095: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -9451> 2013-04-03 10:44:13.357202 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113865 ==== paxos(lease_ack lc 427472 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (1600294790 0 0) 0x2116500 con 0x208a840 -9450> 2013-04-03 10:44:13.357240 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos -9449> 2013-04-03 10:44:13.357256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 -9448> 2013-04-03 10:44:13.357273 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9447> 2013-04-03 10:44:13.368855 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29095 check_down_pgs -9446> 2013-04-03 10:44:13.368884 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29095 v29095: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -9445> 2013-04-03 10:44:13.368910 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -9444> 2013-04-03 10:44:13.369040 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -9443> 2013-04-03 10:44:13.369080 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -9442> 2013-04-03 10:44:13.369111 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos -9441> 2013-04-03 10:44:13.369128 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394090 update_from_paxos version 394090 summary v 394090 -9440> 2013-04-03 10:44:13.369137 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394090 log -9439> 2013-04-03 10:44:13.369153 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9438> 2013-04-03 10:44:13.369172 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -9437> 2013-04-03 10:44:13.392802 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 162 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21817 v187) v2 ==== 289+0+0 (3722924855 0 0) 0x2a51b00 con 0x200d420 -9436> 2013-04-03 10:44:13.392856 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21817 v187) v2 from mds.? 10.252.24.88:6800/17213 -9435> 2013-04-03 10:44:13.392867 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21817 v187) v2 -- ?+0 0x2f7a8c0 con 0x200d420 -9434> 2013-04-03 10:44:14.345150 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394090 encode_pending v394091 -9433> 2013-04-03 10:44:14.345180 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427472) queue_proposal bl 443 bytes; ctx = 0x2c3db40 -9432> 2013-04-03 10:44:14.345188 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427472) propose_queued 427473 443 bytes -9431> 2013-04-03 10:44:14.346427 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427472 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3083a00 -9430> 2013-04-03 10:44:14.375065 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113866 ==== paxos(accept lc 427472 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (162258568 0 0) 0x3083a00 con 0x208a840 -9429> 2013-04-03 10:44:14.375346 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427473 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116500 -9428> 2013-04-03 10:44:14.375375 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427473 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3083c80 -9427> 2013-04-03 10:44:14.375421 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos -9426> 2013-04-03 10:44:14.375449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394090 -9425> 2013-04-03 10:44:14.375465 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos latest full 393798 -9424> 2013-04-03 10:44:14.375483 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394091 update_from_paxos applying incremental log 394091 2013-04-03 10:44:13.345492 mon.0 10.252.24.25:6789/0 161 : [INF] pgmap v29095: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -9423> 2013-04-03 10:44:14.375531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 check_subs -9422> 2013-04-03 10:44:14.375563 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 create_pending v 394092 -9421> 2013-04-03 10:44:14.375585 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394091 _updated_log for mon.0 10.252.24.25:6789/0 -9420> 2013-04-03 10:44:14.375598 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 161) v1 -- ?+0 0x2e8b8c0 con 0x1bac160 -9419> 2013-04-03 10:44:14.375650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos -9418> 2013-04-03 10:44:14.375665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394091 -9417> 2013-04-03 10:44:14.375684 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9416> 2013-04-03 10:44:14.375706 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 161) v1 ==== 0+0+0 (0 0 0) 0x2e8b8c0 con 0x1bac160 -9415> 2013-04-03 10:44:14.375714 7fcdffcdb700 10 handle_log_ack log(last 161) v1 -9414> 2013-04-03 10:44:14.375716 7fcdffcdb700 10 logged 2013-04-03 10:44:13.345492 mon.0 10.252.24.25:6789/0 161 : [INF] pgmap v29095: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1092B/s wr, 0op/s -9413> 2013-04-03 10:44:14.400297 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113867 ==== paxos(lease_ack lc 427473 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (2907580306 0 0) 0x3083c80 con 0x208a840 -9412> 2013-04-03 10:44:14.400337 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos -9411> 2013-04-03 10:44:14.400353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394091 -9410> 2013-04-03 10:44:14.400370 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9409> 2013-04-03 10:44:15.594528 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 163 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140447 v187) v2 ==== 289+0+0 (1934253123 0 0) 0x2f7ab80 con 0x2118420 -9408> 2013-04-03 10:44:15.594613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140447 v187) v2 from mds.? 10.252.58.31:6800/10908 -9407> 2013-04-03 10:44:15.594627 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140447 v187) v2 -- ?+0 0x2a51b00 con 0x2118420 -9406> 2013-04-03 10:44:16.941006 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113868 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 162 v187) v2) to leader v1 ==== 560+0+0 (1149497940 0 0) 0x2a528c0 con 0x208a840 -9405> 2013-04-03 10:44:16.941027 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -9404> 2013-04-03 10:44:16.941038 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc5340 from 10.252.24.88:6789/0 -9403> 2013-04-03 10:44:16.941093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 162 v187) v2 from mds.0 10.252.24.25:6802/3144 -9402> 2013-04-03 10:44:16.941104 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 162 v187) v2 tid 17693) v2 -- ?+0 0x24f8000 con 0x208a840 -9401> 2013-04-03 10:44:17.375453 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427473 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3083780 -9400> 2013-04-03 10:44:17.376185 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113869 ==== paxos(lease_ack lc 427473 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (1686374549 0 0) 0x3083780 con 0x208a840 -9399> 2013-04-03 10:44:17.376261 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos -9398> 2013-04-03 10:44:17.376282 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394091 -9397> 2013-04-03 10:44:17.376299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9396> 2013-04-03 10:44:17.392921 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 163 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21818 v187) v2 ==== 289+0+0 (3092550320 0 0) 0x2a512c0 con 0x200d420 -9395> 2013-04-03 10:44:17.392966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21818 v187) v2 from mds.? 10.252.24.88:6800/17213 -9394> 2013-04-03 10:44:17.392977 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21818 v187) v2 -- ?+0 0x2a528c0 con 0x200d420 -9393> 2013-04-03 10:44:18.267216 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113870 ==== forward(pg_stats(1 pgs tid 31 v 605) v1) to leader v1 ==== 757+0+0 (3584749916 0 0) 0x2f7ab80 con 0x208a840 -9392> 2013-04-03 10:44:18.267244 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -9391> 2013-04-03 10:44:18.267254 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 -9390> 2013-04-03 10:44:18.267305 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 preprocess_query pg_stats(1 pgs tid 31 v 605) v1 from osd.2 10.252.24.30:6801/5656 -9389> 2013-04-03 10:44:18.267329 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 prepare_update pg_stats(1 pgs tid 31 v 605) v1 from osd.2 10.252.24.30:6801/5656 -9388> 2013-04-03 10:44:18.267339 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 prepare_pg_stats pg_stats(1 pgs tid 31 v 605) v1 from osd.2 -9387> 2013-04-03 10:44:18.267345 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29095 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -9386> 2013-04-03 10:44:18.317539 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29095 encode_pending v 29096 -9385> 2013-04-03 10:44:18.317576 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427473) queue_proposal bl 654 bytes; ctx = 0x2c3d9f0 -9384> 2013-04-03 10:44:18.317585 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427473) propose_queued 427474 654 bytes -9383> 2013-04-03 10:44:18.318719 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427473 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3083500 -9382> 2013-04-03 10:44:18.346295 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113871 ==== paxos(accept lc 427473 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1937798500 0 0) 0x3083500 con 0x208a840 -9381> 2013-04-03 10:44:18.346764 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427474 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3083780 -9380> 2013-04-03 10:44:18.346790 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427474 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3083c80 -9379> 2013-04-03 10:44:18.346858 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29095 update_from_paxos applying incremental 29096 -9378> 2013-04-03 10:44:18.346872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 v29096: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1406B/s wr, 0op/s -9377> 2013-04-03 10:44:18.346892 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 send_pg_creates to 0 pgs -9376> 2013-04-03 10:44:18.346894 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 update_logger -9375> 2013-04-03 10:44:18.346904 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 create_pending v 29097 -9374> 2013-04-03 10:44:18.346912 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29096 _updated_stats for osd.2 10.252.24.30:6801/5656 -9373> 2013-04-03 10:44:18.346919 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 31) v1 tid 17694) v2 -- ?+0 0x24f8d80 con 0x208a840 -9372> 2013-04-03 10:44:18.346932 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 check_osd_map already seen 605 >= 605 -9371> 2013-04-03 10:44:18.346936 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 update_logger -9370> 2013-04-03 10:44:18.346946 7fcdffcdb700 0 log [INF] : pgmap v29096: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1406B/s wr, 0op/s -9369> 2013-04-03 10:44:18.346948 7fcdffcdb700 10 send_log to self -9368> 2013-04-03 10:44:18.346948 7fcdffcdb700 10 log_queue is 1 last_log 162 sent 161 num 1 unsent 1 sending 1 -9367> 2013-04-03 10:44:18.346950 7fcdffcdb700 10 will send 2013-04-03 10:44:18.346947 mon.0 10.252.24.25:6789/0 162 : [INF] pgmap v29096: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1406B/s wr, 0op/s -9366> 2013-04-03 10:44:18.346956 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73600 -9365> 2013-04-03 10:44:18.347013 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos -9364> 2013-04-03 10:44:18.347032 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394091 -9363> 2013-04-03 10:44:18.347051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9362> 2013-04-03 10:44:18.347072 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73600 con 0x1bac160 -9361> 2013-04-03 10:44:18.347093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos -9360> 2013-04-03 10:44:18.347108 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394091 -9359> 2013-04-03 10:44:18.347117 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9358> 2013-04-03 10:44:18.347128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 preprocess_log log(1 entries) v1 from mon.0 -9357> 2013-04-03 10:44:18.347151 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9356> 2013-04-03 10:44:18.347162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 prepare_log log(1 entries) v1 from mon.0 -9355> 2013-04-03 10:44:18.347172 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 logging 2013-04-03 10:44:18.346947 mon.0 10.252.24.25:6789/0 162 : [INF] pgmap v29096: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1406B/s wr, 0op/s -9354> 2013-04-03 10:44:18.359527 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113872 ==== paxos(lease_ack lc 427474 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3551042878 0 0) 0x3083c80 con 0x208a840 -9353> 2013-04-03 10:44:18.359569 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos -9352> 2013-04-03 10:44:18.359586 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394091 -9351> 2013-04-03 10:44:18.359600 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9350> 2013-04-03 10:44:18.369284 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29096 check_down_pgs -9349> 2013-04-03 10:44:18.369324 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29096 v29096: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1406B/s wr, 0op/s -9348> 2013-04-03 10:44:18.369349 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -9347> 2013-04-03 10:44:18.369369 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -9346> 2013-04-03 10:44:18.369405 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -9345> 2013-04-03 10:44:18.369436 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos -9344> 2013-04-03 10:44:18.369453 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394091 update_from_paxos version 394091 summary v 394091 -9343> 2013-04-03 10:44:18.369462 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394091 log -9342> 2013-04-03 10:44:18.369477 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9341> 2013-04-03 10:44:18.369497 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -9340> 2013-04-03 10:44:19.346501 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394091 encode_pending v394092 -9339> 2013-04-03 10:44:19.346529 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427474) queue_proposal bl 443 bytes; ctx = 0x2c3da40 -9338> 2013-04-03 10:44:19.346537 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427474) propose_queued 427475 443 bytes -9337> 2013-04-03 10:44:19.347953 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427474 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3083280 -9336> 2013-04-03 10:44:19.433718 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113873 ==== paxos(accept lc 427474 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2420477073 0 0) 0x3083280 con 0x208a840 -9335> 2013-04-03 10:44:19.434029 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427475 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3083c80 -9334> 2013-04-03 10:44:19.434062 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427475 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3083500 -9333> 2013-04-03 10:44:19.434103 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos -9332> 2013-04-03 10:44:19.434134 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos version 394092 summary v 394091 -9331> 2013-04-03 10:44:19.434148 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos latest full 393798 -9330> 2013-04-03 10:44:19.434167 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394092 update_from_paxos applying incremental log 394092 2013-04-03 10:44:18.346947 mon.0 10.252.24.25:6789/0 162 : [INF] pgmap v29096: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1406B/s wr, 0op/s -9329> 2013-04-03 10:44:19.434216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 check_subs -9328> 2013-04-03 10:44:19.434247 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 create_pending v 394093 -9327> 2013-04-03 10:44:19.434269 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394092 _updated_log for mon.0 10.252.24.25:6789/0 -9326> 2013-04-03 10:44:19.434282 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 162) v1 -- ?+0 0x2e8ba80 con 0x1bac160 -9325> 2013-04-03 10:44:19.434334 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos -9324> 2013-04-03 10:44:19.434350 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos version 394092 summary v 394092 -9323> 2013-04-03 10:44:19.434368 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9322> 2013-04-03 10:44:19.434390 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 162) v1 ==== 0+0+0 (0 0 0) 0x2e8ba80 con 0x1bac160 -9321> 2013-04-03 10:44:19.434398 7fcdffcdb700 10 handle_log_ack log(last 162) v1 -9320> 2013-04-03 10:44:19.434401 7fcdffcdb700 10 logged 2013-04-03 10:44:18.346947 mon.0 10.252.24.25:6789/0 162 : [INF] pgmap v29096: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1406B/s wr, 0op/s -9319> 2013-04-03 10:44:19.445636 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113874 ==== paxos(lease_ack lc 427475 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (4246808654 0 0) 0x3083500 con 0x208a840 -9318> 2013-04-03 10:44:19.445720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos -9317> 2013-04-03 10:44:19.445740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos version 394092 summary v 394092 -9316> 2013-04-03 10:44:19.445758 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9315> 2013-04-03 10:44:19.594652 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 164 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140448 v187) v2 ==== 289+0+0 (3264201599 0 0) 0x2a52b80 con 0x2118420 -9314> 2013-04-03 10:44:19.594704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140448 v187) v2 from mds.? 10.252.58.31:6800/10908 -9313> 2013-04-03 10:44:19.594719 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140448 v187) v2 -- ?+0 0x2f7ab80 con 0x2118420 -9312> 2013-04-03 10:44:20.941170 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113875 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 163 v187) v2) to leader v1 ==== 560+0+0 (2583099137 0 0) 0x2cc4dc0 con 0x208a840 -9311> 2013-04-03 10:44:20.941191 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -9310> 2013-04-03 10:44:20.941201 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4840 from 10.252.24.88:6789/0 -9309> 2013-04-03 10:44:20.941256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 163 v187) v2 from mds.0 10.252.24.25:6802/3144 -9308> 2013-04-03 10:44:20.941267 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 163 v187) v2 tid 17695) v2 -- ?+0 0x24f8000 con 0x208a840 -9307> 2013-04-03 10:44:21.049222 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3003500 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33117/0 -9306> 2013-04-03 10:44:21.049245 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003500 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33117/0) -9305> 2013-04-03 10:44:21.049260 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003500 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -9304> 2013-04-03 10:44:21.049489 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003500 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -9303> 2013-04-03 10:44:21.393058 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 164 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21819 v187) v2 ==== 289+0+0 (2604281293 0 0) 0x2a52080 con 0x200d420 -9302> 2013-04-03 10:44:21.393136 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21819 v187) v2 from mds.? 10.252.24.88:6800/17213 -9301> 2013-04-03 10:44:21.393148 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21819 v187) v2 -- ?+0 0x2cc4dc0 con 0x200d420 -9300> 2013-04-03 10:44:21.621657 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113876 ==== forward(pg_stats(1 pgs tid 50 v 605) v1) to leader v1 ==== 753+0+0 (4075514124 0 0) 0x2a52b80 con 0x208a840 -9299> 2013-04-03 10:44:21.621676 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -9298> 2013-04-03 10:44:21.621686 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 -9297> 2013-04-03 10:44:21.621742 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 preprocess_query pg_stats(1 pgs tid 50 v 605) v1 from osd.1 10.252.24.25:6805/3572 -9296> 2013-04-03 10:44:21.621766 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 prepare_update pg_stats(1 pgs tid 50 v 605) v1 from osd.1 10.252.24.25:6805/3572 -9295> 2013-04-03 10:44:21.621772 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 prepare_pg_stats pg_stats(1 pgs tid 50 v 605) v1 from osd.1 -9294> 2013-04-03 10:44:21.621778 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29096 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -9293> 2013-04-03 10:44:21.672332 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29096 encode_pending v 29097 -9292> 2013-04-03 10:44:21.672373 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427475) queue_proposal bl 650 bytes; ctx = 0x2c3db20 -9291> 2013-04-03 10:44:21.672383 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427475) propose_queued 427476 650 bytes -9290> 2013-04-03 10:44:21.673392 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427475 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3083000 -9289> 2013-04-03 10:44:21.735296 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113877 ==== paxos(accept lc 427475 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3140777634 0 0) 0x3083000 con 0x208a840 -9288> 2013-04-03 10:44:21.780223 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427476 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3083500 -9287> 2013-04-03 10:44:21.780248 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427476 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3083280 -9286> 2013-04-03 10:44:21.780318 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29096 update_from_paxos applying incremental 29097 -9285> 2013-04-03 10:44:21.780338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 v29097: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -9284> 2013-04-03 10:44:21.780356 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 send_pg_creates to 0 pgs -9283> 2013-04-03 10:44:21.780360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 update_logger -9282> 2013-04-03 10:44:21.780370 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 create_pending v 29098 -9281> 2013-04-03 10:44:21.780378 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29097 _updated_stats for osd.1 10.252.24.25:6805/3572 -9280> 2013-04-03 10:44:21.780385 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 50) v1 tid 17696) v2 -- ?+0 0x24f8d80 con 0x208a840 -9279> 2013-04-03 10:44:21.780395 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 check_osd_map already seen 605 >= 605 -9278> 2013-04-03 10:44:21.780398 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 update_logger -9277> 2013-04-03 10:44:21.780408 7fcdffcdb700 0 log [INF] : pgmap v29097: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -9276> 2013-04-03 10:44:21.780410 7fcdffcdb700 10 send_log to self -9275> 2013-04-03 10:44:21.780411 7fcdffcdb700 10 log_queue is 1 last_log 163 sent 162 num 1 unsent 1 sending 1 -9274> 2013-04-03 10:44:21.780412 7fcdffcdb700 10 will send 2013-04-03 10:44:21.780409 mon.0 10.252.24.25:6789/0 163 : [INF] pgmap v29097: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -9273> 2013-04-03 10:44:21.780419 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b73600 -9272> 2013-04-03 10:44:21.780469 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos -9271> 2013-04-03 10:44:21.780488 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos version 394092 summary v 394092 -9270> 2013-04-03 10:44:21.780510 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9269> 2013-04-03 10:44:21.780531 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b73600 con 0x1bac160 -9268> 2013-04-03 10:44:21.780552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos -9267> 2013-04-03 10:44:21.780567 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos version 394092 summary v 394092 -9266> 2013-04-03 10:44:21.780576 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9265> 2013-04-03 10:44:21.780588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 preprocess_log log(1 entries) v1 from mon.0 -9264> 2013-04-03 10:44:21.780612 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9263> 2013-04-03 10:44:21.780624 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 prepare_log log(1 entries) v1 from mon.0 -9262> 2013-04-03 10:44:21.780633 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 logging 2013-04-03 10:44:21.780409 mon.0 10.252.24.25:6789/0 163 : [INF] pgmap v29097: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -9261> 2013-04-03 10:44:21.788848 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113878 ==== paxos(lease_ack lc 427476 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (373715059 0 0) 0x3083280 con 0x208a840 -9260> 2013-04-03 10:44:21.788895 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos -9259> 2013-04-03 10:44:21.788913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394092 update_from_paxos version 394092 summary v 394092 -9258> 2013-04-03 10:44:21.788929 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9257> 2013-04-03 10:44:21.789221 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113879 ==== forward(pg_stats(1 pgs tid 30457 v 605) v1) to leader v1 ==== 757+0+0 (4281397406 0 0) 0x2cc4b00 con 0x208a840 -9256> 2013-04-03 10:44:21.789231 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -9255> 2013-04-03 10:44:21.789239 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -9254> 2013-04-03 10:44:21.789264 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 preprocess_query pg_stats(1 pgs tid 30457 v 605) v1 from osd.3 10.252.24.88:6801/17538 -9253> 2013-04-03 10:44:21.789283 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 prepare_update pg_stats(1 pgs tid 30457 v 605) v1 from osd.3 10.252.24.88:6801/17538 -9252> 2013-04-03 10:44:21.789289 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 prepare_pg_stats pg_stats(1 pgs tid 30457 v 605) v1 from osd.3 -9251> 2013-04-03 10:44:21.789296 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29097 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -9250> 2013-04-03 10:44:22.735494 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29097 encode_pending v 29098 -9249> 2013-04-03 10:44:22.735529 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427476) queue_proposal bl 654 bytes; ctx = 0x2c3db30 -9248> 2013-04-03 10:44:22.735538 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427476) propose_queued 427477 654 bytes -9247> 2013-04-03 10:44:22.736556 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427476 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062780 -9246> 2013-04-03 10:44:22.736653 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394092 encode_pending v394093 -9245> 2013-04-03 10:44:22.736683 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427429..427476) queue_proposal bl 442 bytes; ctx = 0x2c3da30 -9244> 2013-04-03 10:44:22.736690 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427429..427476) propose_new_value not active; proposal queued -9243> 2013-04-03 10:44:22.763113 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113880 ==== paxos(accept lc 427476 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (228728210 0 0) 0x3062780 con 0x208a840 -9242> 2013-04-03 10:44:22.763472 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427477 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3083280 -9241> 2013-04-03 10:44:22.763501 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427477 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3083000 -9240> 2013-04-03 10:44:22.763575 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29097 update_from_paxos applying incremental 29098 -9239> 2013-04-03 10:44:22.763594 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 v29098: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s -9238> 2013-04-03 10:44:22.763614 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 send_pg_creates to 0 pgs -9237> 2013-04-03 10:44:22.763619 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 update_logger -9236> 2013-04-03 10:44:22.763629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 create_pending v 29099 -9235> 2013-04-03 10:44:22.763638 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29098 _updated_stats for osd.3 10.252.24.88:6801/17538 -9234> 2013-04-03 10:44:22.763645 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30457) v1 tid 17697) v2 -- ?+0 0x24f8000 con 0x208a840 -9233> 2013-04-03 10:44:22.763655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 check_osd_map already seen 605 >= 605 -9232> 2013-04-03 10:44:22.763658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 update_logger -9231> 2013-04-03 10:44:22.763668 7fcdffcdb700 0 log [INF] : pgmap v29098: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s -9230> 2013-04-03 10:44:22.763669 7fcdffcdb700 10 send_log to self -9229> 2013-04-03 10:44:22.763670 7fcdffcdb700 10 log_queue is 2 last_log 164 sent 163 num 2 unsent 1 sending 1 -9228> 2013-04-03 10:44:22.763672 7fcdffcdb700 10 will send 2013-04-03 10:44:21.780409 mon.0 10.252.24.25:6789/0 163 : [INF] pgmap v29097: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -9227> 2013-04-03 10:44:22.763679 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74800 -9226> 2013-04-03 10:44:22.763703 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427477) propose_queued 427478 442 bytes -9225> 2013-04-03 10:44:22.764411 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427477 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3083a00 -9224> 2013-04-03 10:44:22.764443 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74800 con 0x1bac160 -9223> 2013-04-03 10:44:22.788295 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113881 ==== paxos(lease_ack lc 427477 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (4209935480 0 0) 0x3083a00 con 0x208a840 -9222> 2013-04-03 10:44:22.801381 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113882 ==== paxos(accept lc 427477 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1922066072 0 0) 0x3083000 con 0x208a840 -9221> 2013-04-03 10:44:22.801673 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427478 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3083a00 -9220> 2013-04-03 10:44:22.801707 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427478 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3062780 -9219> 2013-04-03 10:44:22.801741 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos -9218> 2013-04-03 10:44:22.801765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394092 -9217> 2013-04-03 10:44:22.801779 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos latest full 393798 -9216> 2013-04-03 10:44:22.801798 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394093 update_from_paxos applying incremental log 394093 2013-04-03 10:44:21.780409 mon.0 10.252.24.25:6789/0 163 : [INF] pgmap v29097: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -9215> 2013-04-03 10:44:22.801851 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 check_subs -9214> 2013-04-03 10:44:22.801883 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 create_pending v 394094 -9213> 2013-04-03 10:44:22.801905 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394093 _updated_log for mon.0 10.252.24.25:6789/0 -9212> 2013-04-03 10:44:22.801917 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 163) v1 -- ?+0 0x2e8be00 con 0x1bac160 -9211> 2013-04-03 10:44:22.801938 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos -9210> 2013-04-03 10:44:22.801953 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 -9209> 2013-04-03 10:44:22.801962 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -9208> 2013-04-03 10:44:22.801972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 preprocess_log log(1 entries) v1 from mon.0 -9207> 2013-04-03 10:44:22.801989 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 nothing new -9206> 2013-04-03 10:44:22.802040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos -9205> 2013-04-03 10:44:22.802055 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 -9204> 2013-04-03 10:44:22.802074 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9203> 2013-04-03 10:44:22.802095 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 163) v1 ==== 0+0+0 (0 0 0) 0x2e8be00 con 0x1bac160 -9202> 2013-04-03 10:44:22.802103 7fcdffcdb700 10 handle_log_ack log(last 163) v1 -9201> 2013-04-03 10:44:22.802104 7fcdffcdb700 10 logged 2013-04-03 10:44:21.780409 mon.0 10.252.24.25:6789/0 163 : [INF] pgmap v29097: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -9200> 2013-04-03 10:44:22.821783 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113883 ==== paxos(lease_ack lc 427478 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3429909798 0 0) 0x3062780 con 0x208a840 -9199> 2013-04-03 10:44:22.821826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos -9198> 2013-04-03 10:44:22.821843 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 -9197> 2013-04-03 10:44:22.821858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9196> 2013-04-03 10:44:23.369647 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29098 check_down_pgs -9195> 2013-04-03 10:44:23.369674 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29098 v29098: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s -9194> 2013-04-03 10:44:23.369702 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -9193> 2013-04-03 10:44:23.369728 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -9192> 2013-04-03 10:44:23.369765 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -9191> 2013-04-03 10:44:23.369795 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos -9190> 2013-04-03 10:44:23.369814 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 -9189> 2013-04-03 10:44:23.369823 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394093 log -9188> 2013-04-03 10:44:23.369840 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9187> 2013-04-03 10:44:23.369858 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -9186> 2013-04-03 10:44:23.594801 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 165 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140449 v187) v2 ==== 289+0+0 (3789650946 0 0) 0x2a84b80 con 0x2118420 -9185> 2013-04-03 10:44:23.594876 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140449 v187) v2 from mds.? 10.252.58.31:6800/10908 -9184> 2013-04-03 10:44:23.594890 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140449 v187) v2 -- ?+0 0x2cc4b00 con 0x2118420 -9183> 2013-04-03 10:44:24.941264 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113884 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 164 v187) v2) to leader v1 ==== 560+0+0 (715814083 0 0) 0x2cc5b80 con 0x208a840 -9182> 2013-04-03 10:44:24.941284 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -9181> 2013-04-03 10:44:24.941295 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2995b00 from 10.252.24.88:6789/0 -9180> 2013-04-03 10:44:24.941356 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 164 v187) v2 from mds.0 10.252.24.25:6802/3144 -9179> 2013-04-03 10:44:24.941369 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 164 v187) v2 tid 17698) v2 -- ?+0 0x24f8d80 con 0x208a840 -9178> 2013-04-03 10:44:25.393136 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 165 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21820 v187) v2 ==== 289+0+0 (1932726718 0 0) 0x2a51000 con 0x200d420 -9177> 2013-04-03 10:44:25.393217 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21820 v187) v2 from mds.? 10.252.24.88:6800/17213 -9176> 2013-04-03 10:44:25.393230 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21820 v187) v2 -- ?+0 0x2cc5b80 con 0x200d420 -9175> 2013-04-03 10:44:25.801779 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427478 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3062500 -9174> 2013-04-03 10:44:25.802579 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113885 ==== paxos(lease_ack lc 427478 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (3273495582 0 0) 0x3062500 con 0x208a840 -9173> 2013-04-03 10:44:25.802649 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos -9172> 2013-04-03 10:44:25.802669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 -9171> 2013-04-03 10:44:25.802686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9170> 2013-04-03 10:44:27.594936 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 166 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140450 v187) v2 ==== 289+0+0 (2220014981 0 0) 0x2a832c0 con 0x2118420 -9169> 2013-04-03 10:44:27.595011 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140450 v187) v2 from mds.? 10.252.58.31:6800/10908 -9168> 2013-04-03 10:44:27.595024 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140450 v187) v2 -- ?+0 0x2a51000 con 0x2118420 -9167> 2013-04-03 10:44:28.268442 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113886 ==== forward(pg_stats(1 pgs tid 32 v 605) v1) to leader v1 ==== 757+0+0 (1704953071 0 0) 0x2a84b80 con 0x208a840 -9166> 2013-04-03 10:44:28.268464 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -9165> 2013-04-03 10:44:28.268473 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -9164> 2013-04-03 10:44:28.268523 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 preprocess_query pg_stats(1 pgs tid 32 v 605) v1 from osd.2 10.252.24.30:6801/5656 -9163> 2013-04-03 10:44:28.268547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 prepare_update pg_stats(1 pgs tid 32 v 605) v1 from osd.2 10.252.24.30:6801/5656 -9162> 2013-04-03 10:44:28.268553 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 prepare_pg_stats pg_stats(1 pgs tid 32 v 605) v1 from osd.2 -9161> 2013-04-03 10:44:28.268559 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29098 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -9160> 2013-04-03 10:44:28.318745 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29098 encode_pending v 29099 -9159> 2013-04-03 10:44:28.318783 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427478) queue_proposal bl 654 bytes; ctx = 0x2c3d930 -9158> 2013-04-03 10:44:28.318792 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427429..427478) propose_queued 427479 654 bytes -9157> 2013-04-03 10:44:28.320012 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427478 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003780 -9156> 2013-04-03 10:44:28.349463 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113887 ==== paxos(accept lc 427478 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4286900680 0 0) 0x3003780 con 0x208a840 -9155> 2013-04-03 10:44:28.349843 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427479 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062500 -9154> 2013-04-03 10:44:28.349863 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427479 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3062780 -9153> 2013-04-03 10:44:28.349957 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29098 update_from_paxos applying incremental 29099 -9152> 2013-04-03 10:44:28.349978 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29099 v29099: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1386B/s wr, 0op/s -9151> 2013-04-03 10:44:28.349997 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29099 send_pg_creates to 0 pgs -9150> 2013-04-03 10:44:28.350002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29099 update_logger -9149> 2013-04-03 10:44:28.350013 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29099 create_pending v 29100 -9148> 2013-04-03 10:44:28.350022 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29099 _updated_stats for osd.2 10.252.24.30:6801/5656 -9147> 2013-04-03 10:44:28.350030 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 32) v1 tid 17699) v2 -- ?+0 0x24f8000 con 0x208a840 -9146> 2013-04-03 10:44:28.350040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29099 check_osd_map already seen 605 >= 605 -9145> 2013-04-03 10:44:28.350044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29099 update_logger -9144> 2013-04-03 10:44:28.350054 7fcdffcdb700 0 log [INF] : pgmap v29099: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1386B/s wr, 0op/s -9143> 2013-04-03 10:44:28.350056 7fcdffcdb700 10 send_log to self -9142> 2013-04-03 10:44:28.350057 7fcdffcdb700 10 log_queue is 2 last_log 165 sent 163 num 2 unsent 2 sending 2 -9141> 2013-04-03 10:44:28.350059 7fcdffcdb700 10 will send 2013-04-03 10:44:22.763669 mon.0 10.252.24.25:6789/0 164 : [INF] pgmap v29098: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s -9140> 2013-04-03 10:44:28.350065 7fcdffcdb700 10 will send 2013-04-03 10:44:28.350056 mon.0 10.252.24.25:6789/0 165 : [INF] pgmap v29099: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1386B/s wr, 0op/s -9139> 2013-04-03 10:44:28.350652 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b74800 -9138> 2013-04-03 10:44:28.350747 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos -9137> 2013-04-03 10:44:28.350767 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 -9136> 2013-04-03 10:44:28.350784 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9135> 2013-04-03 10:44:28.350807 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74800 con 0x1bac160 -9134> 2013-04-03 10:44:28.350834 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos -9133> 2013-04-03 10:44:28.350849 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 -9132> 2013-04-03 10:44:28.351115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -9131> 2013-04-03 10:44:28.351137 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 preprocess_log log(2 entries) v1 from mon.0 -9130> 2013-04-03 10:44:28.351170 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -9129> 2013-04-03 10:44:28.351183 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 prepare_log log(2 entries) v1 from mon.0 -9128> 2013-04-03 10:44:28.351194 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 logging 2013-04-03 10:44:22.763669 mon.0 10.252.24.25:6789/0 164 : [INF] pgmap v29098: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s -9127> 2013-04-03 10:44:28.351216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 logging 2013-04-03 10:44:28.350056 mon.0 10.252.24.25:6789/0 165 : [INF] pgmap v29099: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1386B/s wr, 0op/s -9126> 2013-04-03 10:44:28.362682 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113888 ==== paxos(lease_ack lc 427479 fc 427429 pn 0 opn 0) v3 ==== 80+0+0 (1299838650 0 0) 0x3062780 con 0x208a840 -9125> 2013-04-03 10:44:28.362754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos -9124> 2013-04-03 10:44:28.362773 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 -9123> 2013-04-03 10:44:28.362789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9122> 2013-04-03 10:44:28.369982 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29099 check_down_pgs -9121> 2013-04-03 10:44:28.370011 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29099 v29099: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1386B/s wr, 0op/s -9120> 2013-04-03 10:44:28.370050 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -9119> 2013-04-03 10:44:28.370072 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -9118> 2013-04-03 10:44:28.370108 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -9117> 2013-04-03 10:44:28.370138 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos -9116> 2013-04-03 10:44:28.370155 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394093 update_from_paxos version 394093 summary v 394093 -9115> 2013-04-03 10:44:28.370166 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394093 log -9114> 2013-04-03 10:44:28.370182 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -9113> 2013-04-03 10:44:28.370199 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -9112> 2013-04-03 10:44:28.759685 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427429..427479) queue_proposal bl 1257 bytes; ctx = 0x2c3db10 -9111> 2013-04-03 10:44:28.815414 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113889 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (424405211 0 0) 0x2996600 con 0x208a840 -9110> 2013-04-03 10:44:28.815432 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -9109> 2013-04-03 10:44:28.815439 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -9108> 2013-04-03 10:44:28.815447 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 -9107> 2013-04-03 10:44:28.815452 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( start_reply flags( reply-to(mon.2 10.252.24.88:6789/0) ) ) v1 -- ?+0 0x2a84b80 -9106> 2013-04-03 10:44:28.941357 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113890 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 165 v187) v2) to leader v1 ==== 560+0+0 (3261211567 0 0) 0x2996b80 con 0x208a840 -9105> 2013-04-03 10:44:28.941372 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -9104> 2013-04-03 10:44:28.941384 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29952c0 from 10.252.24.88:6789/0 -9103> 2013-04-03 10:44:28.941436 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 165 v187) v2 from mds.0 10.252.24.25:6802/3144 -9102> 2013-04-03 10:44:28.941448 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 165 v187) v2 tid 17700) v2 -- ?+0 0x24f8d80 con 0x208a840 -9101> 2013-04-03 10:44:29.349665 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394093 encode_pending v394094 -9100> 2013-04-03 10:44:29.349694 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427429..427479) queue_proposal bl 726 bytes; ctx = 0x2c3d140 -9099> 2013-04-03 10:44:29.349700 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427429..427479) propose_queued 427480 1257 bytes -9098> 2013-04-03 10:44:29.350916 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427479 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003c80 -9097> 2013-04-03 10:44:29.378788 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113891 ==== paxos(accept lc 427479 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2512228233 0 0) 0x3003c80 con 0x208a840 -9096> 2013-04-03 10:44:29.379233 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427480 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062780 -9095> 2013-04-03 10:44:29.379255 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427480 fc 427429 pn 0 opn 0) v3 -- ?+0 0x3003780 -9094> 2013-04-03 10:44:29.379328 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427480) propose_queued 427481 726 bytes -9093> 2013-04-03 10:44:29.379552 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427480 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3083000 -9092> 2013-04-03 10:44:29.390735 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113892 ==== paxos(lease_ack lc 427480 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3777373092 0 0) 0x3083000 con 0x208a840 -9091> 2013-04-03 10:44:29.393184 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 166 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21821 v187) v2 ==== 289+0+0 (1348316867 0 0) 0x2a51dc0 con 0x200d420 -9090> 2013-04-03 10:44:29.393244 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21821 v187) v2 from mds.? 10.252.24.88:6800/17213 -9089> 2013-04-03 10:44:29.393257 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21821 v187) v2 -- ?+0 0x2996b80 con 0x200d420 -9088> 2013-04-03 10:44:29.402540 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113893 ==== paxos(accept lc 427480 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (321175290 0 0) 0x3003780 con 0x208a840 -9087> 2013-04-03 10:44:29.402839 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427481 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3083000 -9086> 2013-04-03 10:44:29.402866 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427481 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3003c80 -9085> 2013-04-03 10:44:29.402900 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos -9084> 2013-04-03 10:44:29.402927 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394093 -9083> 2013-04-03 10:44:29.402944 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos latest full 393798 -9082> 2013-04-03 10:44:29.402964 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos applying incremental log 394094 2013-04-03 10:44:22.763669 mon.0 10.252.24.25:6789/0 164 : [INF] pgmap v29098: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s -9081> 2013-04-03 10:44:29.402997 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos applying incremental log 394094 2013-04-03 10:44:28.350056 mon.0 10.252.24.25:6789/0 165 : [INF] pgmap v29099: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1386B/s wr, 0op/s -9080> 2013-04-03 10:44:29.403046 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 check_subs -9079> 2013-04-03 10:44:29.403080 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 create_pending v 394095 -9078> 2013-04-03 10:44:29.403103 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 _updated_log for mon.0 10.252.24.25:6789/0 -9077> 2013-04-03 10:44:29.403117 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 165) v1 -- ?+0 0x22241c0 con 0x1bac160 -9076> 2013-04-03 10:44:29.403166 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos -9075> 2013-04-03 10:44:29.403182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 -9074> 2013-04-03 10:44:29.403202 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9073> 2013-04-03 10:44:29.403224 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 165) v1 ==== 0+0+0 (0 0 0) 0x22241c0 con 0x1bac160 -9072> 2013-04-03 10:44:29.403232 7fcdffcdb700 10 handle_log_ack log(last 165) v1 -9071> 2013-04-03 10:44:29.403234 7fcdffcdb700 10 logged 2013-04-03 10:44:22.763669 mon.0 10.252.24.25:6789/0 164 : [INF] pgmap v29098: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1158B/s wr, 0op/s -9070> 2013-04-03 10:44:29.403241 7fcdffcdb700 10 logged 2013-04-03 10:44:28.350056 mon.0 10.252.24.25:6789/0 165 : [INF] pgmap v29099: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1386B/s wr, 0op/s -9069> 2013-04-03 10:44:29.428311 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113894 ==== paxos(lease_ack lc 427481 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (2992381761 0 0) 0x3003c80 con 0x208a840 -9068> 2013-04-03 10:44:29.428352 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos -9067> 2013-04-03 10:44:29.428370 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 -9066> 2013-04-03 10:44:29.428385 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9065> 2013-04-03 10:44:31.595079 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 167 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140451 v187) v2 ==== 289+0+0 (2805727992 0 0) 0x2ce2340 con 0x2118420 -9064> 2013-04-03 10:44:31.595153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140451 v187) v2 from mds.? 10.252.58.31:6800/10908 -9063> 2013-04-03 10:44:31.595167 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140451 v187) v2 -- ?+0 0x2a51dc0 con 0x2118420 -9062> 2013-04-03 10:44:32.402942 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427481 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3003500 -9061> 2013-04-03 10:44:32.403708 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113895 ==== paxos(lease_ack lc 427481 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (1320848747 0 0) 0x3003500 con 0x208a840 -9060> 2013-04-03 10:44:32.403777 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos -9059> 2013-04-03 10:44:32.403798 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 -9058> 2013-04-03 10:44:32.403816 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9057> 2013-04-03 10:44:32.941482 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113896 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 166 v187) v2) to leader v1 ==== 560+0+0 (1100010446 0 0) 0x2996600 con 0x208a840 -9056> 2013-04-03 10:44:32.941502 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -9055> 2013-04-03 10:44:32.941513 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2995580 from 10.252.24.88:6789/0 -9054> 2013-04-03 10:44:32.941561 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 166 v187) v2 from mds.0 10.252.24.25:6802/3144 -9053> 2013-04-03 10:44:32.941573 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 166 v187) v2 tid 17701) v2 -- ?+0 0x2b26b40 con 0x208a840 -9052> 2013-04-03 10:44:33.370317 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29099 v29099: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1386B/s wr, 0op/s -9051> 2013-04-03 10:44:33.370360 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -9050> 2013-04-03 10:44:33.370382 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -9049> 2013-04-03 10:44:33.370427 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -9048> 2013-04-03 10:44:33.370459 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos -9047> 2013-04-03 10:44:33.370478 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 -9046> 2013-04-03 10:44:33.370488 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 log -9045> 2013-04-03 10:44:33.370505 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9044> 2013-04-03 10:44:33.370524 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -9043> 2013-04-03 10:44:33.393362 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 167 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21822 v187) v2 ==== 289+0+0 (904818500 0 0) 0x21452c0 con 0x200d420 -9042> 2013-04-03 10:44:33.393403 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21822 v187) v2 from mds.? 10.252.24.88:6800/17213 -9041> 2013-04-03 10:44:33.393413 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21822 v187) v2 -- ?+0 0x2996600 con 0x200d420 -9040> 2013-04-03 10:44:35.403013 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427481 fc 427469 pn 0 opn 0) v3 -- ?+0 0x303a780 -9039> 2013-04-03 10:44:35.403836 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113897 ==== paxos(lease_ack lc 427481 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (803407137 0 0) 0x303a780 con 0x208a840 -9038> 2013-04-03 10:44:35.403905 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos -9037> 2013-04-03 10:44:35.403926 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 -9036> 2013-04-03 10:44:35.403944 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9035> 2013-04-03 10:44:35.595372 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 168 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140452 v187) v2 ==== 289+0+0 (1328889483 0 0) 0x30c2000 con 0x2118420 -9034> 2013-04-03 10:44:35.595450 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140452 v187) v2 from mds.? 10.252.58.31:6800/10908 -9033> 2013-04-03 10:44:35.595465 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140452 v187) v2 -- ?+0 0x21452c0 con 0x2118420 -9032> 2013-04-03 10:44:36.623396 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113898 ==== forward(pg_stats(1 pgs tid 51 v 605) v1) to leader v1 ==== 753+0+0 (2666124781 0 0) 0x2ce2340 con 0x208a840 -9031> 2013-04-03 10:44:36.623418 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -9030> 2013-04-03 10:44:36.623428 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 -9029> 2013-04-03 10:44:36.623484 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29099 preprocess_query pg_stats(1 pgs tid 51 v 605) v1 from osd.1 10.252.24.25:6805/3572 -9028> 2013-04-03 10:44:36.623509 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29099 prepare_update pg_stats(1 pgs tid 51 v 605) v1 from osd.1 10.252.24.25:6805/3572 -9027> 2013-04-03 10:44:36.623515 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29099 prepare_pg_stats pg_stats(1 pgs tid 51 v 605) v1 from osd.1 -9026> 2013-04-03 10:44:36.623522 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29099 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -9025> 2013-04-03 10:44:36.673706 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29099 encode_pending v 29100 -9024> 2013-04-03 10:44:36.673745 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427481) queue_proposal bl 650 bytes; ctx = 0x2c3d9c0 -9023> 2013-04-03 10:44:36.673755 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427481) propose_queued 427482 650 bytes -9022> 2013-04-03 10:44:36.675061 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427481 fc 0 pn 17600 opn 0) v3 -- ?+0 0x303a000 -9021> 2013-04-03 10:44:36.704453 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113899 ==== paxos(accept lc 427481 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4105588490 0 0) 0x303a000 con 0x208a840 -9020> 2013-04-03 10:44:36.704756 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427482 fc 0 pn 17600 opn 0) v3 -- ?+0 0x303a780 -9019> 2013-04-03 10:44:36.704789 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427482 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3003500 -9018> 2013-04-03 10:44:36.704855 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29099 update_from_paxos applying incremental 29100 -9017> 2013-04-03 10:44:36.704873 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 v29100: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1175B/s wr, 0op/s -9016> 2013-04-03 10:44:36.704893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 send_pg_creates to 0 pgs -9015> 2013-04-03 10:44:36.704897 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 update_logger -9014> 2013-04-03 10:44:36.704908 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 create_pending v 29101 -9013> 2013-04-03 10:44:36.704918 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 _updated_stats for osd.1 10.252.24.25:6805/3572 -9012> 2013-04-03 10:44:36.704925 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 51) v1 tid 17702) v2 -- ?+0 0x24f8d80 con 0x208a840 -9011> 2013-04-03 10:44:36.704934 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 check_osd_map already seen 605 >= 605 -9010> 2013-04-03 10:44:36.704938 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 update_logger -9009> 2013-04-03 10:44:36.704955 7fcdffcdb700 0 log [INF] : pgmap v29100: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1175B/s wr, 0op/s -9008> 2013-04-03 10:44:36.704958 7fcdffcdb700 10 send_log to self -9007> 2013-04-03 10:44:36.704959 7fcdffcdb700 10 log_queue is 1 last_log 166 sent 165 num 1 unsent 1 sending 1 -9006> 2013-04-03 10:44:36.704961 7fcdffcdb700 10 will send 2013-04-03 10:44:36.704958 mon.0 10.252.24.25:6789/0 166 : [INF] pgmap v29100: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1175B/s wr, 0op/s -9005> 2013-04-03 10:44:36.704967 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74800 -9004> 2013-04-03 10:44:36.705018 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos -9003> 2013-04-03 10:44:36.705038 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 -9002> 2013-04-03 10:44:36.705057 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -9001> 2013-04-03 10:44:36.705079 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74800 con 0x1bac160 -9000> 2013-04-03 10:44:36.705101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos -8999> 2013-04-03 10:44:36.705117 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 -8998> 2013-04-03 10:44:36.705126 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8997> 2013-04-03 10:44:36.705138 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 preprocess_log log(1 entries) v1 from mon.0 -8996> 2013-04-03 10:44:36.705164 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8995> 2013-04-03 10:44:36.705177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 prepare_log log(1 entries) v1 from mon.0 -8994> 2013-04-03 10:44:36.705189 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 logging 2013-04-03 10:44:36.704958 mon.0 10.252.24.25:6789/0 166 : [INF] pgmap v29100: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1175B/s wr, 0op/s -8993> 2013-04-03 10:44:36.716462 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113900 ==== paxos(lease_ack lc 427482 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3303054107 0 0) 0x3003500 con 0x208a840 -8992> 2013-04-03 10:44:36.716499 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos -8991> 2013-04-03 10:44:36.716516 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 update_from_paxos version 394094 summary v 394094 -8990> 2013-04-03 10:44:36.716533 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8989> 2013-04-03 10:44:36.941621 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113901 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 167 v187) v2) to leader v1 ==== 560+0+0 (2632308379 0 0) 0x2996340 con 0x208a840 -8988> 2013-04-03 10:44:36.941643 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -8987> 2013-04-03 10:44:36.941653 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x29968c0 from 10.252.24.88:6789/0 -8986> 2013-04-03 10:44:36.941705 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 167 v187) v2 from mds.0 10.252.24.25:6802/3144 -8985> 2013-04-03 10:44:36.941717 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 167 v187) v2 tid 17703) v2 -- ?+0 0x2b26b40 con 0x208a840 -8984> 2013-04-03 10:44:37.393407 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 168 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21823 v187) v2 ==== 289+0+0 (377540665 0 0) 0x2145dc0 con 0x200d420 -8983> 2013-04-03 10:44:37.393484 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21823 v187) v2 from mds.? 10.252.24.88:6800/17213 -8982> 2013-04-03 10:44:37.393498 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21823 v187) v2 -- ?+0 0x2996340 con 0x200d420 -8981> 2013-04-03 10:44:37.704936 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394094 encode_pending v394095 -8980> 2013-04-03 10:44:37.704968 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427482) queue_proposal bl 443 bytes; ctx = 0x2c3daf0 -8979> 2013-04-03 10:44:37.704976 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427482) propose_queued 427483 443 bytes -8978> 2013-04-03 10:44:37.706579 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427482 fc 0 pn 17600 opn 0) v3 -- ?+0 0x303ac80 -8977> 2013-04-03 10:44:37.732327 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113902 ==== paxos(accept lc 427482 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3255127576 0 0) 0x303ac80 con 0x208a840 -8976> 2013-04-03 10:44:37.737477 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427483 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003500 -8975> 2013-04-03 10:44:37.737503 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427483 fc 427469 pn 0 opn 0) v3 -- ?+0 0x303a000 -8974> 2013-04-03 10:44:37.737544 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos -8973> 2013-04-03 10:44:37.737572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394094 -8972> 2013-04-03 10:44:37.737587 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos latest full 393798 -8971> 2013-04-03 10:44:37.737606 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos applying incremental log 394095 2013-04-03 10:44:36.704958 mon.0 10.252.24.25:6789/0 166 : [INF] pgmap v29100: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1175B/s wr, 0op/s -8970> 2013-04-03 10:44:37.737664 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 check_subs -8969> 2013-04-03 10:44:37.737698 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 create_pending v 394096 -8968> 2013-04-03 10:44:37.737722 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 _updated_log for mon.0 10.252.24.25:6789/0 -8967> 2013-04-03 10:44:37.737739 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 166) v1 -- ?+0 0x2224c40 con 0x1bac160 -8966> 2013-04-03 10:44:37.737791 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos -8965> 2013-04-03 10:44:37.737808 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 -8964> 2013-04-03 10:44:37.737835 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8963> 2013-04-03 10:44:37.737857 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 166) v1 ==== 0+0+0 (0 0 0) 0x2224c40 con 0x1bac160 -8962> 2013-04-03 10:44:37.737866 7fcdffcdb700 10 handle_log_ack log(last 166) v1 -8961> 2013-04-03 10:44:37.737868 7fcdffcdb700 10 logged 2013-04-03 10:44:36.704958 mon.0 10.252.24.25:6789/0 166 : [INF] pgmap v29100: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1175B/s wr, 0op/s -8960> 2013-04-03 10:44:37.765906 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113903 ==== paxos(lease_ack lc 427483 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (1309038878 0 0) 0x303a000 con 0x208a840 -8959> 2013-04-03 10:44:37.765941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos -8958> 2013-04-03 10:44:37.765958 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 -8957> 2013-04-03 10:44:37.765975 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8956> 2013-04-03 10:44:38.370651 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 check_down_pgs -8955> 2013-04-03 10:44:38.370687 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 v29100: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1175B/s wr, 0op/s -8954> 2013-04-03 10:44:38.370714 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -8953> 2013-04-03 10:44:38.370738 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -8952> 2013-04-03 10:44:38.370776 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -8951> 2013-04-03 10:44:38.370807 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos -8950> 2013-04-03 10:44:38.370825 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 -8949> 2013-04-03 10:44:38.370835 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 log -8948> 2013-04-03 10:44:38.370852 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8947> 2013-04-03 10:44:38.370870 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -8946> 2013-04-03 10:44:39.595311 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 169 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140453 v187) v2 ==== 289+0+0 (1817936374 0 0) 0x2ce1580 con 0x2118420 -8945> 2013-04-03 10:44:39.595388 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140453 v187) v2 from mds.? 10.252.58.31:6800/10908 -8944> 2013-04-03 10:44:39.595402 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140453 v187) v2 -- ?+0 0x2145dc0 con 0x2118420 -8943> 2013-04-03 10:44:40.737578 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427483 fc 427469 pn 0 opn 0) v3 -- ?+0 0x303aa00 -8942> 2013-04-03 10:44:40.738378 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113904 ==== paxos(lease_ack lc 427483 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (804074732 0 0) 0x303aa00 con 0x208a840 -8941> 2013-04-03 10:44:40.738446 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos -8940> 2013-04-03 10:44:40.738467 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 -8939> 2013-04-03 10:44:40.738485 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8938> 2013-04-03 10:44:40.941689 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113905 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 168 v187) v2) to leader v1 ==== 560+0+0 (3897965706 0 0) 0x2ce2340 con 0x208a840 -8937> 2013-04-03 10:44:40.941705 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -8936> 2013-04-03 10:44:40.941714 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ef9580 from 10.252.24.88:6789/0 -8935> 2013-04-03 10:44:40.941748 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 168 v187) v2 from mds.0 10.252.24.25:6802/3144 -8934> 2013-04-03 10:44:40.941760 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 168 v187) v2 tid 17704) v2 -- ?+0 0x24f8d80 con 0x208a840 -8933> 2013-04-03 10:44:41.081248 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3003280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33121/0 -8932> 2013-04-03 10:44:41.081267 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33121/0) -8931> 2013-04-03 10:44:41.081282 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003280 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -8930> 2013-04-03 10:44:41.081515 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003280 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -8929> 2013-04-03 10:44:41.393569 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 169 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21824 v187) v2 ==== 289+0+0 (1401497549 0 0) 0x2145840 con 0x200d420 -8928> 2013-04-03 10:44:41.393614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21824 v187) v2 from mds.? 10.252.24.88:6800/17213 -8927> 2013-04-03 10:44:41.393624 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21824 v187) v2 -- ?+0 0x2ce2340 con 0x200d420 -8926> 2013-04-03 10:44:43.371013 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 v29100: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1175B/s wr, 0op/s -8925> 2013-04-03 10:44:43.371055 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -8924> 2013-04-03 10:44:43.371081 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -8923> 2013-04-03 10:44:43.371125 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -8922> 2013-04-03 10:44:43.371156 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos -8921> 2013-04-03 10:44:43.371176 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 -8920> 2013-04-03 10:44:43.371185 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 log -8919> 2013-04-03 10:44:43.371202 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8918> 2013-04-03 10:44:43.371222 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -8917> 2013-04-03 10:44:43.595420 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 170 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140454 v187) v2 ==== 289+0+0 (166215793 0 0) 0x2ce12c0 con 0x2118420 -8916> 2013-04-03 10:44:43.595493 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140454 v187) v2 from mds.? 10.252.58.31:6800/10908 -8915> 2013-04-03 10:44:43.595506 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140454 v187) v2 -- ?+0 0x2145840 con 0x2118420 -8914> 2013-04-03 10:44:43.737642 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427483 fc 427469 pn 0 opn 0) v3 -- ?+0 0x300c000 -8913> 2013-04-03 10:44:43.738445 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113906 ==== paxos(lease_ack lc 427483 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3528462531 0 0) 0x300c000 con 0x208a840 -8912> 2013-04-03 10:44:43.738491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos -8911> 2013-04-03 10:44:43.738511 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 -8910> 2013-04-03 10:44:43.738529 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8909> 2013-04-03 10:44:44.941896 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113907 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 169 v187) v2) to leader v1 ==== 560+0+0 (3776963209 0 0) 0x2ce1580 con 0x208a840 -8908> 2013-04-03 10:44:44.941916 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -8907> 2013-04-03 10:44:44.941926 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ef9840 from 10.252.24.88:6789/0 -8906> 2013-04-03 10:44:44.941982 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 169 v187) v2 from mds.0 10.252.24.25:6802/3144 -8905> 2013-04-03 10:44:44.941994 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 169 v187) v2 tid 17705) v2 -- ?+0 0x2b26b40 con 0x208a840 -8904> 2013-04-03 10:44:45.393651 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 170 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21825 v187) v2 ==== 289+0+0 (1894226096 0 0) 0x2146080 con 0x200d420 -8903> 2013-04-03 10:44:45.393699 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21825 v187) v2 from mds.? 10.252.24.88:6800/17213 -8902> 2013-04-03 10:44:45.393709 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21825 v187) v2 -- ?+0 0x2ce1580 con 0x200d420 -8901> 2013-04-03 10:44:46.737715 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427483 fc 427469 pn 0 opn 0) v3 -- ?+0 0x300c780 -8900> 2013-04-03 10:44:46.738469 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113908 ==== paxos(lease_ack lc 427483 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (41863495 0 0) 0x300c780 con 0x208a840 -8899> 2013-04-03 10:44:46.738538 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos -8898> 2013-04-03 10:44:46.738559 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 -8897> 2013-04-03 10:44:46.738577 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8896> 2013-04-03 10:44:47.595579 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 171 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140455 v187) v2 ==== 289+0+0 (713490188 0 0) 0x2ce28c0 con 0x2118420 -8895> 2013-04-03 10:44:47.595657 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140455 v187) v2 from mds.? 10.252.58.31:6800/10908 -8894> 2013-04-03 10:44:47.595674 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140455 v187) v2 -- ?+0 0x2146080 con 0x2118420 -8893> 2013-04-03 10:44:48.271031 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113909 ==== forward(pg_stats(1 pgs tid 33 v 605) v1) to leader v1 ==== 757+0+0 (1760975852 0 0) 0x2ce12c0 con 0x208a840 -8892> 2013-04-03 10:44:48.271050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -8891> 2013-04-03 10:44:48.271059 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 -8890> 2013-04-03 10:44:48.271103 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 preprocess_query pg_stats(1 pgs tid 33 v 605) v1 from osd.2 10.252.24.30:6801/5656 -8889> 2013-04-03 10:44:48.271127 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 prepare_update pg_stats(1 pgs tid 33 v 605) v1 from osd.2 10.252.24.30:6801/5656 -8888> 2013-04-03 10:44:48.271136 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 prepare_pg_stats pg_stats(1 pgs tid 33 v 605) v1 from osd.2 -8887> 2013-04-03 10:44:48.271143 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -8886> 2013-04-03 10:44:48.321328 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 encode_pending v 29101 -8885> 2013-04-03 10:44:48.321366 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427483) queue_proposal bl 654 bytes; ctx = 0x2c3d990 -8884> 2013-04-03 10:44:48.321376 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427483) propose_queued 427484 654 bytes -8883> 2013-04-03 10:44:48.322897 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427483 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300c280 -8882> 2013-04-03 10:44:48.348217 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113910 ==== paxos(accept lc 427483 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3166410676 0 0) 0x300c280 con 0x208a840 -8881> 2013-04-03 10:44:48.353789 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427484 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300c780 -8880> 2013-04-03 10:44:48.353822 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427484 fc 427469 pn 0 opn 0) v3 -- ?+0 0x300c000 -8879> 2013-04-03 10:44:48.353894 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29100 update_from_paxos applying incremental 29101 -8878> 2013-04-03 10:44:48.353911 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 v29101: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -8877> 2013-04-03 10:44:48.353931 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 send_pg_creates to 0 pgs -8876> 2013-04-03 10:44:48.353935 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 update_logger -8875> 2013-04-03 10:44:48.353946 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 create_pending v 29102 -8874> 2013-04-03 10:44:48.353958 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 _updated_stats for osd.2 10.252.24.30:6801/5656 -8873> 2013-04-03 10:44:48.353965 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 33) v1 tid 17706) v2 -- ?+0 0x24f8d80 con 0x208a840 -8872> 2013-04-03 10:44:48.353975 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 check_osd_map already seen 605 >= 605 -8871> 2013-04-03 10:44:48.353978 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 update_logger -8870> 2013-04-03 10:44:48.353989 7fcdffcdb700 0 log [INF] : pgmap v29101: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -8869> 2013-04-03 10:44:48.353991 7fcdffcdb700 10 send_log to self -8868> 2013-04-03 10:44:48.353992 7fcdffcdb700 10 log_queue is 1 last_log 167 sent 166 num 1 unsent 1 sending 1 -8867> 2013-04-03 10:44:48.353994 7fcdffcdb700 10 will send 2013-04-03 10:44:48.353990 mon.0 10.252.24.25:6789/0 167 : [INF] pgmap v29101: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -8866> 2013-04-03 10:44:48.354000 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74800 -8865> 2013-04-03 10:44:48.354054 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos -8864> 2013-04-03 10:44:48.354075 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 -8863> 2013-04-03 10:44:48.354094 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8862> 2013-04-03 10:44:48.354115 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74800 con 0x1bac160 -8861> 2013-04-03 10:44:48.354137 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos -8860> 2013-04-03 10:44:48.354153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 -8859> 2013-04-03 10:44:48.354162 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8858> 2013-04-03 10:44:48.354174 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 preprocess_log log(1 entries) v1 from mon.0 -8857> 2013-04-03 10:44:48.354200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8856> 2013-04-03 10:44:48.354213 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 prepare_log log(1 entries) v1 from mon.0 -8855> 2013-04-03 10:44:48.354223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 logging 2013-04-03 10:44:48.353990 mon.0 10.252.24.25:6789/0 167 : [INF] pgmap v29101: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -8854> 2013-04-03 10:44:48.368515 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113911 ==== paxos(lease_ack lc 427484 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (4214388449 0 0) 0x300c000 con 0x208a840 -8853> 2013-04-03 10:44:48.368551 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos -8852> 2013-04-03 10:44:48.368567 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 -8851> 2013-04-03 10:44:48.368584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8850> 2013-04-03 10:44:48.371332 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 check_down_pgs -8849> 2013-04-03 10:44:48.371362 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 v29101: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -8848> 2013-04-03 10:44:48.371392 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -8847> 2013-04-03 10:44:48.371691 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -8846> 2013-04-03 10:44:48.371732 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -8845> 2013-04-03 10:44:48.371764 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos -8844> 2013-04-03 10:44:48.371782 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 update_from_paxos version 394095 summary v 394095 -8843> 2013-04-03 10:44:48.371791 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 log -8842> 2013-04-03 10:44:48.371808 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8841> 2013-04-03 10:44:48.371825 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -8840> 2013-04-03 10:44:48.941992 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113912 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 170 v187) v2) to leader v1 ==== 560+0+0 (3067201982 0 0) 0x2ef9b00 con 0x208a840 -8839> 2013-04-03 10:44:48.942014 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -8838> 2013-04-03 10:44:48.942024 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2efa8c0 from 10.252.24.88:6789/0 -8837> 2013-04-03 10:44:48.942078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 170 v187) v2 from mds.0 10.252.24.25:6802/3144 -8836> 2013-04-03 10:44:48.942092 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 170 v187) v2 tid 17707) v2 -- ?+0 0x2b26b40 con 0x208a840 -8835> 2013-04-03 10:44:49.348412 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394095 encode_pending v394096 -8834> 2013-04-03 10:44:49.348442 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427484) queue_proposal bl 443 bytes; ctx = 0x2c3d9b0 -8833> 2013-04-03 10:44:49.348450 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427484) propose_queued 427485 443 bytes -8832> 2013-04-03 10:44:49.349525 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427484 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300ca00 -8831> 2013-04-03 10:44:49.376048 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113913 ==== paxos(accept lc 427484 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2326116678 0 0) 0x300ca00 con 0x208a840 -8830> 2013-04-03 10:44:49.377004 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427485 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300c000 -8829> 2013-04-03 10:44:49.377036 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427485 fc 427469 pn 0 opn 0) v3 -- ?+0 0x300c280 -8828> 2013-04-03 10:44:49.377073 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos -8827> 2013-04-03 10:44:49.377098 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394095 -8826> 2013-04-03 10:44:49.377114 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos latest full 393798 -8825> 2013-04-03 10:44:49.377133 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos applying incremental log 394096 2013-04-03 10:44:48.353990 mon.0 10.252.24.25:6789/0 167 : [INF] pgmap v29101: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -8824> 2013-04-03 10:44:49.377180 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 check_subs -8823> 2013-04-03 10:44:49.377212 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 create_pending v 394097 -8822> 2013-04-03 10:44:49.377235 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 _updated_log for mon.0 10.252.24.25:6789/0 -8821> 2013-04-03 10:44:49.377248 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 167) v1 -- ?+0 0x28a5c40 con 0x1bac160 -8820> 2013-04-03 10:44:49.377301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos -8819> 2013-04-03 10:44:49.377328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 -8818> 2013-04-03 10:44:49.377348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8817> 2013-04-03 10:44:49.377371 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 167) v1 ==== 0+0+0 (0 0 0) 0x28a5c40 con 0x1bac160 -8816> 2013-04-03 10:44:49.377380 7fcdffcdb700 10 handle_log_ack log(last 167) v1 -8815> 2013-04-03 10:44:49.377381 7fcdffcdb700 10 logged 2013-04-03 10:44:48.353990 mon.0 10.252.24.25:6789/0 167 : [INF] pgmap v29101: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -8814> 2013-04-03 10:44:49.393749 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 171 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21826 v187) v2 ==== 289+0+0 (357849399 0 0) 0x2145b00 con 0x200d420 -8813> 2013-04-03 10:44:49.393796 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21826 v187) v2 from mds.? 10.252.24.88:6800/17213 -8812> 2013-04-03 10:44:49.393808 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21826 v187) v2 -- ?+0 0x2ef9b00 con 0x200d420 -8811> 2013-04-03 10:44:49.401260 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113914 ==== paxos(lease_ack lc 427485 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (956133825 0 0) 0x300c280 con 0x208a840 -8810> 2013-04-03 10:44:49.401303 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos -8809> 2013-04-03 10:44:49.401328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 -8808> 2013-04-03 10:44:49.401345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8807> 2013-04-03 10:44:51.097194 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3003000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33122/0 -8806> 2013-04-03 10:44:51.097219 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33122/0) -8805> 2013-04-03 10:44:51.097233 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003000 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -8804> 2013-04-03 10:44:51.097446 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003000 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -8803> 2013-04-03 10:44:51.595678 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 172 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140456 v187) v2 ==== 289+0+0 (3692449382 0 0) 0x2ce2b80 con 0x2118420 -8802> 2013-04-03 10:44:51.595758 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140456 v187) v2 from mds.? 10.252.58.31:6800/10908 -8801> 2013-04-03 10:44:51.595772 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140456 v187) v2 -- ?+0 0x2145b00 con 0x2118420 -8800> 2013-04-03 10:44:52.377111 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427485 fc 427469 pn 0 opn 0) v3 -- ?+0 0x300cc80 -8799> 2013-04-03 10:44:52.377837 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113915 ==== paxos(lease_ack lc 427485 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (536099022 0 0) 0x300cc80 con 0x208a840 -8798> 2013-04-03 10:44:52.377911 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos -8797> 2013-04-03 10:44:52.377932 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 -8796> 2013-04-03 10:44:52.377949 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8795> 2013-04-03 10:44:52.942081 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113916 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 171 v187) v2) to leader v1 ==== 560+0+0 (302897120 0 0) 0x2ce12c0 con 0x208a840 -8794> 2013-04-03 10:44:52.942102 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -8793> 2013-04-03 10:44:52.942113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ef92c0 from 10.252.24.88:6789/0 -8792> 2013-04-03 10:44:52.942160 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 171 v187) v2 from mds.0 10.252.24.25:6802/3144 -8791> 2013-04-03 10:44:52.942172 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 171 v187) v2 tid 17708) v2 -- ?+0 0x24f8d80 con 0x208a840 -8790> 2013-04-03 10:44:53.371948 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 v29101: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -8789> 2013-04-03 10:44:53.371990 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -8788> 2013-04-03 10:44:53.372011 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -8787> 2013-04-03 10:44:53.372055 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -8786> 2013-04-03 10:44:53.372086 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos -8785> 2013-04-03 10:44:53.372106 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 -8784> 2013-04-03 10:44:53.372238 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 log -8783> 2013-04-03 10:44:53.372261 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8782> 2013-04-03 10:44:53.372282 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -8781> 2013-04-03 10:44:53.393866 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 172 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21827 v187) v2 ==== 289+0+0 (909829706 0 0) 0x2145580 con 0x200d420 -8780> 2013-04-03 10:44:53.393917 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21827 v187) v2 from mds.? 10.252.24.88:6800/17213 -8779> 2013-04-03 10:44:53.393928 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21827 v187) v2 -- ?+0 0x2ce12c0 con 0x200d420 -8778> 2013-04-03 10:44:55.377182 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427485 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2fe4000 -8777> 2013-04-03 10:44:55.378064 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113917 ==== paxos(lease_ack lc 427485 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (1414646670 0 0) 0x2fe4000 con 0x208a840 -8776> 2013-04-03 10:44:55.378139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos -8775> 2013-04-03 10:44:55.378160 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 -8774> 2013-04-03 10:44:55.378178 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8773> 2013-04-03 10:44:55.595795 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 173 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140457 v187) v2 ==== 289+0+0 (4286116123 0 0) 0x2ce2080 con 0x2118420 -8772> 2013-04-03 10:44:55.595871 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140457 v187) v2 from mds.? 10.252.58.31:6800/10908 -8771> 2013-04-03 10:44:55.595886 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140457 v187) v2 -- ?+0 0x2145580 con 0x2118420 -8770> 2013-04-03 10:44:56.625860 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113918 ==== forward(pg_stats(1 pgs tid 52 v 605) v1) to leader v1 ==== 753+0+0 (3236540356 0 0) 0x2ce2b80 con 0x208a840 -8769> 2013-04-03 10:44:56.625881 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -8768> 2013-04-03 10:44:56.625893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -8767> 2013-04-03 10:44:56.625945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 preprocess_query pg_stats(1 pgs tid 52 v 605) v1 from osd.1 10.252.24.25:6805/3572 -8766> 2013-04-03 10:44:56.625970 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 prepare_update pg_stats(1 pgs tid 52 v 605) v1 from osd.1 10.252.24.25:6805/3572 -8765> 2013-04-03 10:44:56.625977 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 prepare_pg_stats pg_stats(1 pgs tid 52 v 605) v1 from osd.1 -8764> 2013-04-03 10:44:56.625984 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -8763> 2013-04-03 10:44:56.676192 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 encode_pending v 29102 -8762> 2013-04-03 10:44:56.676232 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427485) queue_proposal bl 650 bytes; ctx = 0x2c3dc90 -8761> 2013-04-03 10:44:56.676242 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427485) propose_queued 427486 650 bytes -8760> 2013-04-03 10:44:56.677846 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427485 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4780 -8759> 2013-04-03 10:44:56.946110 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113919 ==== paxos(accept lc 427485 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (307836158 0 0) 0x2fe4780 con 0x208a840 -8758> 2013-04-03 10:44:56.946422 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427486 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4000 -8757> 2013-04-03 10:44:56.946450 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427486 fc 427469 pn 0 opn 0) v3 -- ?+0 0x300cc80 -8756> 2013-04-03 10:44:56.946526 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29101 update_from_paxos applying incremental 29102 -8755> 2013-04-03 10:44:56.946545 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 v29102: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s -8754> 2013-04-03 10:44:56.946565 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 send_pg_creates to 0 pgs -8753> 2013-04-03 10:44:56.946569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 update_logger -8752> 2013-04-03 10:44:56.946580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 create_pending v 29103 -8751> 2013-04-03 10:44:56.946590 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 _updated_stats for osd.1 10.252.24.25:6805/3572 -8750> 2013-04-03 10:44:56.946598 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 52) v1 tid 17709) v2 -- ?+0 0x2b26b40 con 0x208a840 -8749> 2013-04-03 10:44:56.946608 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 check_osd_map already seen 605 >= 605 -8748> 2013-04-03 10:44:56.946612 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 update_logger -8747> 2013-04-03 10:44:56.946623 7fcdffcdb700 0 log [INF] : pgmap v29102: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s -8746> 2013-04-03 10:44:56.946624 7fcdffcdb700 10 send_log to self -8745> 2013-04-03 10:44:56.946625 7fcdffcdb700 10 log_queue is 1 last_log 168 sent 167 num 1 unsent 1 sending 1 -8744> 2013-04-03 10:44:56.946627 7fcdffcdb700 10 will send 2013-04-03 10:44:56.946624 mon.0 10.252.24.25:6789/0 168 : [INF] pgmap v29102: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s -8743> 2013-04-03 10:44:56.946635 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74800 -8742> 2013-04-03 10:44:56.946689 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos -8741> 2013-04-03 10:44:56.946709 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 -8740> 2013-04-03 10:44:56.946728 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8739> 2013-04-03 10:44:56.946750 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74800 con 0x1bac160 -8738> 2013-04-03 10:44:56.946772 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos -8737> 2013-04-03 10:44:56.946790 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 -8736> 2013-04-03 10:44:56.946799 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8735> 2013-04-03 10:44:56.946812 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 preprocess_log log(1 entries) v1 from mon.0 -8734> 2013-04-03 10:44:56.946837 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8733> 2013-04-03 10:44:56.946850 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 prepare_log log(1 entries) v1 from mon.0 -8732> 2013-04-03 10:44:56.946860 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 logging 2013-04-03 10:44:56.946624 mon.0 10.252.24.25:6789/0 168 : [INF] pgmap v29102: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s -8731> 2013-04-03 10:44:56.958026 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113920 ==== paxos(lease_ack lc 427486 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (1412146170 0 0) 0x300cc80 con 0x208a840 -8730> 2013-04-03 10:44:56.958061 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos -8729> 2013-04-03 10:44:56.958078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 update_from_paxos version 394096 summary v 394096 -8728> 2013-04-03 10:44:56.958096 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8727> 2013-04-03 10:44:56.958321 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113921 ==== forward(pg_stats(1 pgs tid 30458 v 605) v1) to leader v1 ==== 757+0+0 (3653726145 0 0) 0x2efa080 con 0x208a840 -8726> 2013-04-03 10:44:56.958369 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -8725> 2013-04-03 10:44:56.958378 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 -8724> 2013-04-03 10:44:56.958406 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 preprocess_query pg_stats(1 pgs tid 30458 v 605) v1 from osd.3 10.252.24.88:6801/17538 -8723> 2013-04-03 10:44:56.958427 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 prepare_update pg_stats(1 pgs tid 30458 v 605) v1 from osd.3 10.252.24.88:6801/17538 -8722> 2013-04-03 10:44:56.958433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 prepare_pg_stats pg_stats(1 pgs tid 30458 v 605) v1 from osd.3 -8721> 2013-04-03 10:44:56.958439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -8720> 2013-04-03 10:44:56.958461 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113922 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 172 v187) v2) to leader v1 ==== 560+0+0 (424517584 0 0) 0x2efa600 con 0x208a840 -8719> 2013-04-03 10:44:56.958471 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -8718> 2013-04-03 10:44:56.958479 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ef9dc0 from 10.252.24.88:6789/0 -8717> 2013-04-03 10:44:56.958505 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 172 v187) v2 from mds.0 10.252.24.25:6802/3144 -8716> 2013-04-03 10:44:56.958516 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 172 v187) v2 tid 17711) v2 -- ?+0 0x24f8d80 con 0x208a840 -8715> 2013-04-03 10:44:57.394000 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 173 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21828 v187) v2 ==== 289+0+0 (3727940153 0 0) 0x2f792c0 con 0x200d420 -8714> 2013-04-03 10:44:57.394080 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21828 v187) v2 from mds.? 10.252.24.88:6800/17213 -8713> 2013-04-03 10:44:57.394094 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21828 v187) v2 -- ?+0 0x2efa600 con 0x200d420 -8712> 2013-04-03 10:44:57.946308 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 encode_pending v 29103 -8711> 2013-04-03 10:44:57.946343 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427486) queue_proposal bl 654 bytes; ctx = 0x2c3dad0 -8710> 2013-04-03 10:44:57.946353 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427486) propose_queued 427487 654 bytes -8709> 2013-04-03 10:44:57.947296 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427486 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4c80 -8708> 2013-04-03 10:44:57.947403 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394096 encode_pending v394097 -8707> 2013-04-03 10:44:57.947431 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427469..427486) queue_proposal bl 443 bytes; ctx = 0x2c3dae0 -8706> 2013-04-03 10:44:57.947438 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427469..427486) propose_new_value not active; proposal queued -8705> 2013-04-03 10:44:57.990566 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113923 ==== paxos(accept lc 427486 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (430264315 0 0) 0x2fe4c80 con 0x208a840 -8704> 2013-04-03 10:44:57.990844 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427487 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300cc80 -8703> 2013-04-03 10:44:57.990871 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427487 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2fe4780 -8702> 2013-04-03 10:44:57.990947 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29102 update_from_paxos applying incremental 29103 -8701> 2013-04-03 10:44:57.990967 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 v29103: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 531B/s wr, 0op/s -8700> 2013-04-03 10:44:57.990987 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 send_pg_creates to 0 pgs -8699> 2013-04-03 10:44:57.990991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 update_logger -8698> 2013-04-03 10:44:57.991002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 create_pending v 29104 -8697> 2013-04-03 10:44:57.991011 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 _updated_stats for osd.3 10.252.24.88:6801/17538 -8696> 2013-04-03 10:44:57.991019 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30458) v1 tid 17710) v2 -- ?+0 0x24f8b40 con 0x208a840 -8695> 2013-04-03 10:44:57.991030 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 check_osd_map already seen 605 >= 605 -8694> 2013-04-03 10:44:57.991034 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 update_logger -8693> 2013-04-03 10:44:57.991045 7fcdffcdb700 0 log [INF] : pgmap v29103: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 531B/s wr, 0op/s -8692> 2013-04-03 10:44:57.991052 7fcdffcdb700 10 send_log to self -8691> 2013-04-03 10:44:57.991053 7fcdffcdb700 10 log_queue is 2 last_log 169 sent 168 num 2 unsent 1 sending 1 -8690> 2013-04-03 10:44:57.991055 7fcdffcdb700 10 will send 2013-04-03 10:44:56.946624 mon.0 10.252.24.25:6789/0 168 : [INF] pgmap v29102: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s -8689> 2013-04-03 10:44:57.991062 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74a00 -8688> 2013-04-03 10:44:57.991088 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427487) propose_queued 427488 443 bytes -8687> 2013-04-03 10:44:57.991297 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427487 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300c280 -8686> 2013-04-03 10:44:57.991330 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74a00 con 0x1bac160 -8685> 2013-04-03 10:44:58.015801 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113924 ==== paxos(lease_ack lc 427487 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (736697163 0 0) 0x300c280 con 0x208a840 -8684> 2013-04-03 10:44:58.040855 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113925 ==== paxos(accept lc 427487 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1797826666 0 0) 0x2fe4780 con 0x208a840 -8683> 2013-04-03 10:44:58.041900 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427488 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300c280 -8682> 2013-04-03 10:44:58.041918 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427488 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2fe4c80 -8681> 2013-04-03 10:44:58.041955 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos -8680> 2013-04-03 10:44:58.041981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos version 394097 summary v 394096 -8679> 2013-04-03 10:44:58.041995 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos latest full 393798 -8678> 2013-04-03 10:44:58.042015 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos applying incremental log 394097 2013-04-03 10:44:56.946624 mon.0 10.252.24.25:6789/0 168 : [INF] pgmap v29102: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s -8677> 2013-04-03 10:44:58.042060 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 check_subs -8676> 2013-04-03 10:44:58.042092 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 create_pending v 394098 -8675> 2013-04-03 10:44:58.042115 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 _updated_log for mon.0 10.252.24.25:6789/0 -8674> 2013-04-03 10:44:58.042128 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 168) v1 -- ?+0 0x28a51c0 con 0x1bac160 -8673> 2013-04-03 10:44:58.042151 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos -8672> 2013-04-03 10:44:58.042167 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos version 394097 summary v 394097 -8671> 2013-04-03 10:44:58.042177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8670> 2013-04-03 10:44:58.042189 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 preprocess_log log(1 entries) v1 from mon.0 -8669> 2013-04-03 10:44:58.042209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 nothing new -8668> 2013-04-03 10:44:58.042260 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos -8667> 2013-04-03 10:44:58.042276 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos version 394097 summary v 394097 -8666> 2013-04-03 10:44:58.042295 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8665> 2013-04-03 10:44:58.042316 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 168) v1 ==== 0+0+0 (0 0 0) 0x28a51c0 con 0x1bac160 -8664> 2013-04-03 10:44:58.042324 7fcdffcdb700 10 handle_log_ack log(last 168) v1 -8663> 2013-04-03 10:44:58.042326 7fcdffcdb700 10 logged 2013-04-03 10:44:56.946624 mon.0 10.252.24.25:6789/0 168 : [INF] pgmap v29102: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1023B/s wr, 0op/s -8662> 2013-04-03 10:44:58.052810 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113926 ==== paxos(lease_ack lc 427488 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (1450614056 0 0) 0x2fe4c80 con 0x208a840 -8661> 2013-04-03 10:44:58.052964 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos -8660> 2013-04-03 10:44:58.052981 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos version 394097 summary v 394097 -8659> 2013-04-03 10:44:58.052998 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8658> 2013-04-03 10:44:58.272172 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113927 ==== forward(pg_stats(1 pgs tid 34 v 605) v1) to leader v1 ==== 757+0+0 (2536163754 0 0) 0x2efa080 con 0x208a840 -8657> 2013-04-03 10:44:58.272186 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -8656> 2013-04-03 10:44:58.272195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -8655> 2013-04-03 10:44:58.272226 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 preprocess_query pg_stats(1 pgs tid 34 v 605) v1 from osd.2 10.252.24.30:6801/5656 -8654> 2013-04-03 10:44:58.272248 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 prepare_update pg_stats(1 pgs tid 34 v 605) v1 from osd.2 10.252.24.30:6801/5656 -8653> 2013-04-03 10:44:58.272257 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 prepare_pg_stats pg_stats(1 pgs tid 34 v 605) v1 from osd.2 -8652> 2013-04-03 10:44:58.272263 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -8651> 2013-04-03 10:44:58.372393 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 check_down_pgs -8650> 2013-04-03 10:44:58.372422 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29103 v29103: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 531B/s wr, 0op/s -8649> 2013-04-03 10:44:58.372451 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -8648> 2013-04-03 10:44:58.372473 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -8647> 2013-04-03 10:44:58.372515 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -8646> 2013-04-03 10:44:58.372547 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos -8645> 2013-04-03 10:44:58.372564 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 update_from_paxos version 394097 summary v 394097 -8644> 2013-04-03 10:44:58.372574 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394097 log -8643> 2013-04-03 10:44:58.372590 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8642> 2013-04-03 10:44:58.372607 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -8641> 2013-04-03 10:44:58.815522 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) -8640> 2013-04-03 10:44:58.815566 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader -8639> 2013-04-03 10:44:59.041046 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29103 encode_pending v 29104 -8638> 2013-04-03 10:44:59.041083 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427488) queue_proposal bl 654 bytes; ctx = 0x2c3dc70 -8637> 2013-04-03 10:44:59.041092 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427488) propose_queued 427489 654 bytes -8636> 2013-04-03 10:44:59.042718 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427488 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2000 -8635> 2013-04-03 10:44:59.068925 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113928 ==== paxos(accept lc 427488 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2433418960 0 0) 0x2df2000 con 0x208a840 -8634> 2013-04-03 10:44:59.069263 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427489 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4c80 -8633> 2013-04-03 10:44:59.069290 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427489 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2fe4780 -8632> 2013-04-03 10:44:59.069370 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29103 update_from_paxos applying incremental 29104 -8631> 2013-04-03 10:44:59.069389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 v29104: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 5629B/s wr, 0op/s -8630> 2013-04-03 10:44:59.069408 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 send_pg_creates to 0 pgs -8629> 2013-04-03 10:44:59.069411 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 update_logger -8628> 2013-04-03 10:44:59.069421 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 create_pending v 29105 -8627> 2013-04-03 10:44:59.069430 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29104 _updated_stats for osd.2 10.252.24.30:6801/5656 -8626> 2013-04-03 10:44:59.069437 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 34) v1 tid 17712) v2 -- ?+0 0x2b26b40 con 0x208a840 -8625> 2013-04-03 10:44:59.069448 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 check_osd_map already seen 605 >= 605 -8624> 2013-04-03 10:44:59.069451 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 update_logger -8623> 2013-04-03 10:44:59.069461 7fcdffcdb700 0 log [INF] : pgmap v29104: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 5629B/s wr, 0op/s -8622> 2013-04-03 10:44:59.069463 7fcdffcdb700 10 send_log to self -8621> 2013-04-03 10:44:59.069463 7fcdffcdb700 10 log_queue is 2 last_log 170 sent 168 num 2 unsent 2 sending 2 -8620> 2013-04-03 10:44:59.069465 7fcdffcdb700 10 will send 2013-04-03 10:44:57.991052 mon.0 10.252.24.25:6789/0 169 : [INF] pgmap v29103: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 531B/s wr, 0op/s -8619> 2013-04-03 10:44:59.069471 7fcdffcdb700 10 will send 2013-04-03 10:44:59.069462 mon.0 10.252.24.25:6789/0 170 : [INF] pgmap v29104: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 5629B/s wr, 0op/s -8618> 2013-04-03 10:44:59.070555 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b74a00 -8617> 2013-04-03 10:44:59.070638 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 update_from_paxos -8616> 2013-04-03 10:44:59.070658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 update_from_paxos version 394097 summary v 394097 -8615> 2013-04-03 10:44:59.070676 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8614> 2013-04-03 10:44:59.070699 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74a00 con 0x1bac160 -8613> 2013-04-03 10:44:59.070721 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 update_from_paxos -8612> 2013-04-03 10:44:59.070736 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 update_from_paxos version 394097 summary v 394097 -8611> 2013-04-03 10:44:59.070745 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -8610> 2013-04-03 10:44:59.070757 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 preprocess_log log(2 entries) v1 from mon.0 -8609> 2013-04-03 10:44:59.070788 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -8608> 2013-04-03 10:44:59.070801 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 prepare_log log(2 entries) v1 from mon.0 -8607> 2013-04-03 10:44:59.070812 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 logging 2013-04-03 10:44:57.991052 mon.0 10.252.24.25:6789/0 169 : [INF] pgmap v29103: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 531B/s wr, 0op/s -8606> 2013-04-03 10:44:59.070833 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 logging 2013-04-03 10:44:59.069462 mon.0 10.252.24.25:6789/0 170 : [INF] pgmap v29104: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 5629B/s wr, 0op/s -8605> 2013-04-03 10:44:59.094120 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113929 ==== paxos(lease_ack lc 427489 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (1143992329 0 0) 0x2fe4780 con 0x208a840 -8604> 2013-04-03 10:44:59.094157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 update_from_paxos -8603> 2013-04-03 10:44:59.094173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394097 update_from_paxos version 394097 summary v 394097 -8602> 2013-04-03 10:44:59.094190 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8601> 2013-04-03 10:44:59.595935 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 174 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140458 v187) v2 ==== 289+0+0 (2597007516 0 0) 0x2ce1dc0 con 0x2118420 -8600> 2013-04-03 10:44:59.596010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140458 v187) v2 from mds.? 10.252.58.31:6800/10908 -8599> 2013-04-03 10:44:59.596028 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140458 v187) v2 -- ?+0 0x2efa080 con 0x2118420 -8598> 2013-04-03 10:45:00.069127 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394097 encode_pending v394098 -8597> 2013-04-03 10:45:00.069157 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427489) queue_proposal bl 725 bytes; ctx = 0x2c3dab0 -8596> 2013-04-03 10:45:00.069164 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427489) propose_queued 427490 725 bytes -8595> 2013-04-03 10:45:00.070211 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427489 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2a00 -8594> 2013-04-03 10:45:00.092167 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113930 ==== paxos(accept lc 427489 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2560970195 0 0) 0x2df2a00 con 0x208a840 -8593> 2013-04-03 10:45:00.092519 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427490 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4780 -8592> 2013-04-03 10:45:00.092551 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427490 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2df2000 -8591> 2013-04-03 10:45:00.092587 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos -8590> 2013-04-03 10:45:00.092614 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394097 -8589> 2013-04-03 10:45:00.092629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos latest full 393798 -8588> 2013-04-03 10:45:00.093278 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394098 update_from_paxos applying incremental log 394098 2013-04-03 10:44:57.991052 mon.0 10.252.24.25:6789/0 169 : [INF] pgmap v29103: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 531B/s wr, 0op/s -8587> 2013-04-03 10:45:00.093315 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394098 update_from_paxos applying incremental log 394098 2013-04-03 10:44:59.069462 mon.0 10.252.24.25:6789/0 170 : [INF] pgmap v29104: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 5629B/s wr, 0op/s -8586> 2013-04-03 10:45:00.093353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 check_subs -8585> 2013-04-03 10:45:00.093385 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 create_pending v 394099 -8584> 2013-04-03 10:45:00.093407 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394098 _updated_log for mon.0 10.252.24.25:6789/0 -8583> 2013-04-03 10:45:00.093420 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 170) v1 -- ?+0 0x28a5380 con 0x1bac160 -8582> 2013-04-03 10:45:00.093472 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos -8581> 2013-04-03 10:45:00.093488 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 -8580> 2013-04-03 10:45:00.093506 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8579> 2013-04-03 10:45:00.093529 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 170) v1 ==== 0+0+0 (0 0 0) 0x28a5380 con 0x1bac160 -8578> 2013-04-03 10:45:00.093537 7fcdffcdb700 10 handle_log_ack log(last 170) v1 -8577> 2013-04-03 10:45:00.093538 7fcdffcdb700 10 logged 2013-04-03 10:44:57.991052 mon.0 10.252.24.25:6789/0 169 : [INF] pgmap v29103: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 531B/s wr, 0op/s -8576> 2013-04-03 10:45:00.093547 7fcdffcdb700 10 logged 2013-04-03 10:44:59.069462 mon.0 10.252.24.25:6789/0 170 : [INF] pgmap v29104: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 5629B/s wr, 0op/s -8575> 2013-04-03 10:45:00.105438 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113931 ==== paxos(lease_ack lc 427490 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3963865082 0 0) 0x2df2000 con 0x208a840 -8574> 2013-04-03 10:45:00.105477 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos -8573> 2013-04-03 10:45:00.105494 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 -8572> 2013-04-03 10:45:00.105508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8571> 2013-04-03 10:45:00.942344 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113932 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 173 v187) v2) to leader v1 ==== 560+0+0 (1275733034 0 0) 0x2efa340 con 0x208a840 -8570> 2013-04-03 10:45:00.942366 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -8569> 2013-04-03 10:45:00.942376 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2efab80 from 10.252.24.88:6789/0 -8568> 2013-04-03 10:45:00.942428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 173 v187) v2 from mds.0 10.252.24.25:6802/3144 -8567> 2013-04-03 10:45:00.942439 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 173 v187) v2 tid 17713) v2 -- ?+0 0x24f8b40 con 0x208a840 -8566> 2013-04-03 10:45:01.113169 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3062280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33164/0 -8565> 2013-04-03 10:45:01.113188 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3062280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33164/0) -8564> 2013-04-03 10:45:01.113200 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3062280 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -8563> 2013-04-03 10:45:01.113418 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3062280 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -8562> 2013-04-03 10:45:01.394121 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 174 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21829 v187) v2 ==== 289+0+0 (4250756420 0 0) 0x2f7a080 con 0x200d420 -8561> 2013-04-03 10:45:01.394173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21829 v187) v2 from mds.? 10.252.24.88:6800/17213 -8560> 2013-04-03 10:45:01.394183 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21829 v187) v2 -- ?+0 0x2efa340 con 0x200d420 -8559> 2013-04-03 10:45:03.092630 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427490 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2df2280 -8558> 2013-04-03 10:45:03.093412 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113933 ==== paxos(lease_ack lc 427490 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (1768947046 0 0) 0x2df2280 con 0x208a840 -8557> 2013-04-03 10:45:03.093480 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos -8556> 2013-04-03 10:45:03.093500 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 -8555> 2013-04-03 10:45:03.093517 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8554> 2013-04-03 10:45:03.372718 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29104 check_down_pgs -8553> 2013-04-03 10:45:03.372747 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29104 v29104: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 5629B/s wr, 0op/s -8552> 2013-04-03 10:45:03.372774 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -8551> 2013-04-03 10:45:03.372793 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -8550> 2013-04-03 10:45:03.372829 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -8549> 2013-04-03 10:45:03.372860 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos -8548> 2013-04-03 10:45:03.372876 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 -8547> 2013-04-03 10:45:03.372885 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394098 log -8546> 2013-04-03 10:45:03.372900 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8545> 2013-04-03 10:45:03.372917 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -8544> 2013-04-03 10:45:03.596068 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 175 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140459 v187) v2 ==== 289+0+0 (3114635233 0 0) 0x20e4080 con 0x2118420 -8543> 2013-04-03 10:45:03.596140 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140459 v187) v2 from mds.? 10.252.58.31:6800/10908 -8542> 2013-04-03 10:45:03.596157 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140459 v187) v2 -- ?+0 0x2f7a080 con 0x2118420 -8541> 2013-04-03 10:45:04.942474 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113934 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 174 v187) v2) to leader v1 ==== 560+0+0 (1473267247 0 0) 0x2ce1dc0 con 0x208a840 -8540> 2013-04-03 10:45:04.942495 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -8539> 2013-04-03 10:45:04.942505 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83000 from 10.252.24.88:6789/0 -8538> 2013-04-03 10:45:04.942558 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 174 v187) v2 from mds.0 10.252.24.25:6802/3144 -8537> 2013-04-03 10:45:04.942569 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 174 v187) v2 tid 17714) v2 -- ?+0 0x2b26b40 con 0x208a840 -8536> 2013-04-03 10:45:05.394202 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 175 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21830 v187) v2 ==== 289+0+0 (2565776579 0 0) 0x2f79dc0 con 0x200d420 -8535> 2013-04-03 10:45:05.394258 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21830 v187) v2 from mds.? 10.252.24.88:6800/17213 -8534> 2013-04-03 10:45:05.394267 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21830 v187) v2 -- ?+0 0x2ce1dc0 con 0x200d420 -8533> 2013-04-03 10:45:06.092706 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427490 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2df2500 -8532> 2013-04-03 10:45:06.093451 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113935 ==== paxos(lease_ack lc 427490 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (1587298761 0 0) 0x2df2500 con 0x208a840 -8531> 2013-04-03 10:45:06.093518 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos -8530> 2013-04-03 10:45:06.093538 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 -8529> 2013-04-03 10:45:06.093557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8528> 2013-04-03 10:45:06.626933 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113936 ==== forward(pg_stats(1 pgs tid 53 v 605) v1) to leader v1 ==== 753+0+0 (1445592232 0 0) 0x20e4080 con 0x208a840 -8527> 2013-04-03 10:45:06.626954 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -8526> 2013-04-03 10:45:06.626964 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 -8525> 2013-04-03 10:45:06.627016 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 preprocess_query pg_stats(1 pgs tid 53 v 605) v1 from osd.1 10.252.24.25:6805/3572 -8524> 2013-04-03 10:45:06.627041 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 prepare_update pg_stats(1 pgs tid 53 v 605) v1 from osd.1 10.252.24.25:6805/3572 -8523> 2013-04-03 10:45:06.627048 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 prepare_pg_stats pg_stats(1 pgs tid 53 v 605) v1 from osd.1 -8522> 2013-04-03 10:45:06.627056 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29104 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -8521> 2013-04-03 10:45:06.677242 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29104 encode_pending v 29105 -8520> 2013-04-03 10:45:06.677279 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427490) queue_proposal bl 650 bytes; ctx = 0x2c3dc60 -8519> 2013-04-03 10:45:06.677288 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427490) propose_queued 427491 650 bytes -8518> 2013-04-03 10:45:06.678256 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427490 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4780 -8517> 2013-04-03 10:45:06.715944 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113937 ==== paxos(accept lc 427490 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2799231711 0 0) 0x29b4780 con 0x208a840 -8516> 2013-04-03 10:45:06.716786 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427491 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2500 -8515> 2013-04-03 10:45:06.716813 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427491 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2df2280 -8514> 2013-04-03 10:45:06.716879 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29104 update_from_paxos applying incremental 29105 -8513> 2013-04-03 10:45:06.716894 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 v29105: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1876B/s wr, 0op/s -8512> 2013-04-03 10:45:06.716914 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 send_pg_creates to 0 pgs -8511> 2013-04-03 10:45:06.716917 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 update_logger -8510> 2013-04-03 10:45:06.716927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 create_pending v 29106 -8509> 2013-04-03 10:45:06.716936 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29105 _updated_stats for osd.1 10.252.24.25:6805/3572 -8508> 2013-04-03 10:45:06.716943 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 53) v1 tid 17715) v2 -- ?+0 0x24f8b40 con 0x208a840 -8507> 2013-04-03 10:45:06.716952 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 check_osd_map already seen 605 >= 605 -8506> 2013-04-03 10:45:06.716955 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 update_logger -8505> 2013-04-03 10:45:06.716965 7fcdffcdb700 0 log [INF] : pgmap v29105: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1876B/s wr, 0op/s -8504> 2013-04-03 10:45:06.716967 7fcdffcdb700 10 send_log to self -8503> 2013-04-03 10:45:06.716968 7fcdffcdb700 10 log_queue is 1 last_log 171 sent 170 num 1 unsent 1 sending 1 -8502> 2013-04-03 10:45:06.716970 7fcdffcdb700 10 will send 2013-04-03 10:45:06.716966 mon.0 10.252.24.25:6789/0 171 : [INF] pgmap v29105: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1876B/s wr, 0op/s -8501> 2013-04-03 10:45:06.716976 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74a00 -8500> 2013-04-03 10:45:06.717023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos -8499> 2013-04-03 10:45:06.717044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 -8498> 2013-04-03 10:45:06.717063 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8497> 2013-04-03 10:45:06.717084 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74a00 con 0x1bac160 -8496> 2013-04-03 10:45:06.717105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos -8495> 2013-04-03 10:45:06.717120 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 -8494> 2013-04-03 10:45:06.717129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8493> 2013-04-03 10:45:06.717141 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 preprocess_log log(1 entries) v1 from mon.0 -8492> 2013-04-03 10:45:06.717165 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8491> 2013-04-03 10:45:06.717177 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 prepare_log log(1 entries) v1 from mon.0 -8490> 2013-04-03 10:45:06.717186 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 logging 2013-04-03 10:45:06.716966 mon.0 10.252.24.25:6789/0 171 : [INF] pgmap v29105: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1876B/s wr, 0op/s -8489> 2013-04-03 10:45:06.741268 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113938 ==== paxos(lease_ack lc 427491 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (2553726708 0 0) 0x2df2280 con 0x208a840 -8488> 2013-04-03 10:45:06.741315 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos -8487> 2013-04-03 10:45:06.741333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394098 update_from_paxos version 394098 summary v 394098 -8486> 2013-04-03 10:45:06.741349 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8485> 2013-04-03 10:45:07.596177 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 176 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140460 v187) v2 ==== 289+0+0 (1370272658 0 0) 0x20e4b80 con 0x2118420 -8484> 2013-04-03 10:45:07.596251 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140460 v187) v2 from mds.? 10.252.58.31:6800/10908 -8483> 2013-04-03 10:45:07.596264 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140460 v187) v2 -- ?+0 0x20e4080 con 0x2118420 -8482> 2013-04-03 10:45:07.716150 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394098 encode_pending v394099 -8481> 2013-04-03 10:45:07.716178 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427491) queue_proposal bl 443 bytes; ctx = 0x2c3daa0 -8480> 2013-04-03 10:45:07.716186 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427491) propose_queued 427492 443 bytes -8479> 2013-04-03 10:45:07.717188 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427491 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4a00 -8478> 2013-04-03 10:45:07.739290 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113939 ==== paxos(accept lc 427491 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1147329321 0 0) 0x29b4a00 con 0x208a840 -8477> 2013-04-03 10:45:07.739637 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427492 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2280 -8476> 2013-04-03 10:45:07.739664 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427492 fc 427469 pn 0 opn 0) v3 -- ?+0 0x29b4780 -8475> 2013-04-03 10:45:07.739704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos -8474> 2013-04-03 10:45:07.739731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394098 -8473> 2013-04-03 10:45:07.739746 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos latest full 393798 -8472> 2013-04-03 10:45:07.739765 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394099 update_from_paxos applying incremental log 394099 2013-04-03 10:45:06.716966 mon.0 10.252.24.25:6789/0 171 : [INF] pgmap v29105: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1876B/s wr, 0op/s -8471> 2013-04-03 10:45:07.739815 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 check_subs -8470> 2013-04-03 10:45:07.739847 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 create_pending v 394100 -8469> 2013-04-03 10:45:07.739869 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394099 _updated_log for mon.0 10.252.24.25:6789/0 -8468> 2013-04-03 10:45:07.739884 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 171) v1 -- ?+0 0x22241c0 con 0x1bac160 -8467> 2013-04-03 10:45:07.739936 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos -8466> 2013-04-03 10:45:07.739951 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 -8465> 2013-04-03 10:45:07.739969 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8464> 2013-04-03 10:45:07.739991 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 171) v1 ==== 0+0+0 (0 0 0) 0x22241c0 con 0x1bac160 -8463> 2013-04-03 10:45:07.739999 7fcdffcdb700 10 handle_log_ack log(last 171) v1 -8462> 2013-04-03 10:45:07.740001 7fcdffcdb700 10 logged 2013-04-03 10:45:06.716966 mon.0 10.252.24.25:6789/0 171 : [INF] pgmap v29105: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1876B/s wr, 0op/s -8461> 2013-04-03 10:45:07.752571 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113940 ==== paxos(lease_ack lc 427492 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (2642976753 0 0) 0x29b4780 con 0x208a840 -8460> 2013-04-03 10:45:07.752612 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos -8459> 2013-04-03 10:45:07.752629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 -8458> 2013-04-03 10:45:07.752646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8457> 2013-04-03 10:45:08.373060 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29105 check_down_pgs -8456> 2013-04-03 10:45:08.373093 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29105 v29105: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 1876B/s wr, 0op/s -8455> 2013-04-03 10:45:08.373120 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -8454> 2013-04-03 10:45:08.373142 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -8453> 2013-04-03 10:45:08.373179 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -8452> 2013-04-03 10:45:08.373210 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos -8451> 2013-04-03 10:45:08.373229 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 -8450> 2013-04-03 10:45:08.373238 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394099 log -8449> 2013-04-03 10:45:08.373255 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8448> 2013-04-03 10:45:08.373274 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -8447> 2013-04-03 10:45:08.942720 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113941 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 175 v187) v2) to leader v1 ==== 560+0+0 (3206025539 0 0) 0x2a848c0 con 0x208a840 -8446> 2013-04-03 10:45:08.942742 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -8445> 2013-04-03 10:45:08.942751 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83580 from 10.252.24.88:6789/0 -8444> 2013-04-03 10:45:08.942804 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 175 v187) v2 from mds.0 10.252.24.25:6802/3144 -8443> 2013-04-03 10:45:08.942815 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 175 v187) v2 tid 17716) v2 -- ?+0 0x2b26b40 con 0x208a840 -8442> 2013-04-03 10:45:09.394346 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 176 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21831 v187) v2 ==== 289+0+0 (3145735102 0 0) 0x2f79580 con 0x200d420 -8441> 2013-04-03 10:45:09.394424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21831 v187) v2 from mds.? 10.252.24.88:6800/17213 -8440> 2013-04-03 10:45:09.394437 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21831 v187) v2 -- ?+0 0x2a848c0 con 0x200d420 -8439> 2013-04-03 10:45:10.739741 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427492 fc 427469 pn 0 opn 0) v3 -- ?+0 0x29b4000 -8438> 2013-04-03 10:45:10.740625 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113942 ==== paxos(lease_ack lc 427492 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (2902011895 0 0) 0x29b4000 con 0x208a840 -8437> 2013-04-03 10:45:10.740694 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos -8436> 2013-04-03 10:45:10.740714 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 -8435> 2013-04-03 10:45:10.740731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8434> 2013-04-03 10:45:11.129194 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3003280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33165/0 -8433> 2013-04-03 10:45:11.129221 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33165/0) -8432> 2013-04-03 10:45:11.129234 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003280 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -8431> 2013-04-03 10:45:11.129440 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003280 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -8430> 2013-04-03 10:45:11.596356 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 177 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140461 v187) v2 ==== 289+0+0 (1925319919 0 0) 0x20e48c0 con 0x2118420 -8429> 2013-04-03 10:45:11.596434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140461 v187) v2 from mds.? 10.252.58.31:6800/10908 -8428> 2013-04-03 10:45:11.596448 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140461 v187) v2 -- ?+0 0x2f79580 con 0x2118420 -8427> 2013-04-03 10:45:12.942730 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113943 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 176 v187) v2) to leader v1 ==== 560+0+0 (1328297897 0 0) 0x20e4b80 con 0x208a840 -8426> 2013-04-03 10:45:12.942752 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -8425> 2013-04-03 10:45:12.942762 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a84340 from 10.252.24.88:6789/0 -8424> 2013-04-03 10:45:12.942815 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 176 v187) v2 from mds.0 10.252.24.25:6802/3144 -8423> 2013-04-03 10:45:12.942828 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 176 v187) v2 tid 17717) v2 -- ?+0 0x24f8b40 con 0x208a840 -8422> 2013-04-03 10:45:13.273946 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113944 ==== forward(pg_stats(1 pgs tid 35 v 605) v1) to leader v1 ==== 757+0+0 (1589539145 0 0) 0x20e48c0 con 0x208a840 -8421> 2013-04-03 10:45:13.273959 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -8420> 2013-04-03 10:45:13.273967 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -8419> 2013-04-03 10:45:13.274000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 preprocess_query pg_stats(1 pgs tid 35 v 605) v1 from osd.2 10.252.24.30:6801/5656 -8418> 2013-04-03 10:45:13.274023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 prepare_update pg_stats(1 pgs tid 35 v 605) v1 from osd.2 10.252.24.30:6801/5656 -8417> 2013-04-03 10:45:13.274029 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 prepare_pg_stats pg_stats(1 pgs tid 35 v 605) v1 from osd.2 -8416> 2013-04-03 10:45:13.274035 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29105 got osd.2 osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -8415> 2013-04-03 10:45:13.324221 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29105 encode_pending v 29106 -8414> 2013-04-03 10:45:13.324252 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427492) queue_proposal bl 654 bytes; ctx = 0x2c3dc50 -8413> 2013-04-03 10:45:13.324261 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427492) propose_queued 427493 654 bytes -8412> 2013-04-03 10:45:13.325359 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427492 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4c80 -8411> 2013-04-03 10:45:13.355279 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113945 ==== paxos(accept lc 427492 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4195579755 0 0) 0x29b4c80 con 0x208a840 -8410> 2013-04-03 10:45:13.355568 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427493 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4000 -8409> 2013-04-03 10:45:13.355594 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427493 fc 427469 pn 0 opn 0) v3 -- ?+0 0x29b4780 -8408> 2013-04-03 10:45:13.355665 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29105 update_from_paxos applying incremental 29106 -8407> 2013-04-03 10:45:13.355680 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 v29106: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 50471B/s wr, 0op/s -8406> 2013-04-03 10:45:13.355699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 send_pg_creates to 0 pgs -8405> 2013-04-03 10:45:13.355702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 update_logger -8404> 2013-04-03 10:45:13.355714 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 create_pending v 29107 -8403> 2013-04-03 10:45:13.355723 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29106 _updated_stats for osd.2 10.252.24.30:6801/5656 -8402> 2013-04-03 10:45:13.355728 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 35) v1 tid 17718) v2 -- ?+0 0x2b26b40 con 0x208a840 -8401> 2013-04-03 10:45:13.355737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 check_osd_map already seen 605 >= 605 -8400> 2013-04-03 10:45:13.355740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 update_logger -8399> 2013-04-03 10:45:13.355750 7fcdffcdb700 0 log [INF] : pgmap v29106: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 50471B/s wr, 0op/s -8398> 2013-04-03 10:45:13.355752 7fcdffcdb700 10 send_log to self -8397> 2013-04-03 10:45:13.355753 7fcdffcdb700 10 log_queue is 1 last_log 172 sent 171 num 1 unsent 1 sending 1 -8396> 2013-04-03 10:45:13.355754 7fcdffcdb700 10 will send 2013-04-03 10:45:13.355752 mon.0 10.252.24.25:6789/0 172 : [INF] pgmap v29106: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 50471B/s wr, 0op/s -8395> 2013-04-03 10:45:13.355761 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74a00 -8394> 2013-04-03 10:45:13.355810 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos -8393> 2013-04-03 10:45:13.355829 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 -8392> 2013-04-03 10:45:13.355848 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8391> 2013-04-03 10:45:13.355869 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74a00 con 0x1bac160 -8390> 2013-04-03 10:45:13.355891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos -8389> 2013-04-03 10:45:13.355905 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 -8388> 2013-04-03 10:45:13.355914 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8387> 2013-04-03 10:45:13.355926 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 preprocess_log log(1 entries) v1 from mon.0 -8386> 2013-04-03 10:45:13.355951 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8385> 2013-04-03 10:45:13.355965 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 prepare_log log(1 entries) v1 from mon.0 -8384> 2013-04-03 10:45:13.355976 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 logging 2013-04-03 10:45:13.355752 mon.0 10.252.24.25:6789/0 172 : [INF] pgmap v29106: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 50471B/s wr, 0op/s -8383> 2013-04-03 10:45:13.373387 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29106 check_down_pgs -8382> 2013-04-03 10:45:13.373410 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29106 v29106: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 50471B/s wr, 0op/s -8381> 2013-04-03 10:45:13.373431 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -8380> 2013-04-03 10:45:13.373451 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -8379> 2013-04-03 10:45:13.373486 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -8378> 2013-04-03 10:45:13.373516 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos -8377> 2013-04-03 10:45:13.373533 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 -8376> 2013-04-03 10:45:13.373542 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394099 log -8375> 2013-04-03 10:45:13.373565 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8374> 2013-04-03 10:45:13.373583 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -8373> 2013-04-03 10:45:13.380457 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113946 ==== paxos(lease_ack lc 427493 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (2735496737 0 0) 0x29b4780 con 0x208a840 -8372> 2013-04-03 10:45:13.380498 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos -8371> 2013-04-03 10:45:13.380515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394099 update_from_paxos version 394099 summary v 394099 -8370> 2013-04-03 10:45:13.380530 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8369> 2013-04-03 10:45:13.394386 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 177 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21832 v187) v2 ==== 289+0+0 (1292937940 0 0) 0x2f79b00 con 0x200d420 -8368> 2013-04-03 10:45:13.394434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21832 v187) v2 from mds.? 10.252.24.88:6800/17213 -8367> 2013-04-03 10:45:13.394444 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21832 v187) v2 -- ?+0 0x2996080 con 0x200d420 -8366> 2013-04-03 10:45:14.355476 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394099 encode_pending v394100 -8365> 2013-04-03 10:45:14.355504 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427493) queue_proposal bl 444 bytes; ctx = 0x2c3da90 -8364> 2013-04-03 10:45:14.355511 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427493) propose_queued 427494 444 bytes -8363> 2013-04-03 10:45:14.356561 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427493 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4a00 -8362> 2013-04-03 10:45:14.379658 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113947 ==== paxos(accept lc 427493 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1425300875 0 0) 0x20f4a00 con 0x208a840 -8361> 2013-04-03 10:45:14.380682 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427494 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4780 -8360> 2013-04-03 10:45:14.380711 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427494 fc 427469 pn 0 opn 0) v3 -- ?+0 0x29b4c80 -8359> 2013-04-03 10:45:14.380749 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos -8358> 2013-04-03 10:45:14.380776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394099 -8357> 2013-04-03 10:45:14.380791 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos latest full 393798 -8356> 2013-04-03 10:45:14.380808 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394100 update_from_paxos applying incremental log 394100 2013-04-03 10:45:13.355752 mon.0 10.252.24.25:6789/0 172 : [INF] pgmap v29106: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 50471B/s wr, 0op/s -8355> 2013-04-03 10:45:14.380853 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 check_subs -8354> 2013-04-03 10:45:14.380885 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 create_pending v 394101 -8353> 2013-04-03 10:45:14.380907 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394100 _updated_log for mon.0 10.252.24.25:6789/0 -8352> 2013-04-03 10:45:14.380922 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 172) v1 -- ?+0 0x2224000 con 0x1bac160 -8351> 2013-04-03 10:45:14.380975 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos -8350> 2013-04-03 10:45:14.380991 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394100 -8349> 2013-04-03 10:45:14.381009 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8348> 2013-04-03 10:45:14.381032 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 172) v1 ==== 0+0+0 (0 0 0) 0x2224000 con 0x1bac160 -8347> 2013-04-03 10:45:14.381040 7fcdffcdb700 10 handle_log_ack log(last 172) v1 -8346> 2013-04-03 10:45:14.381041 7fcdffcdb700 10 logged 2013-04-03 10:45:13.355752 mon.0 10.252.24.25:6789/0 172 : [INF] pgmap v29106: 240 pgs: 240 active+clean; 255 MB data, 5659 MB used, 1067 GB / 1079 GB avail; 50471B/s wr, 0op/s -8345> 2013-04-03 10:45:14.391644 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113948 ==== paxos(lease_ack lc 427494 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (1448944352 0 0) 0x29b4c80 con 0x208a840 -8344> 2013-04-03 10:45:14.391679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos -8343> 2013-04-03 10:45:14.391695 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394100 -8342> 2013-04-03 10:45:14.391711 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8341> 2013-04-03 10:45:15.596447 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 178 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140462 v187) v2 ==== 289+0+0 (393334120 0 0) 0x20e3dc0 con 0x2118420 -8340> 2013-04-03 10:45:15.596524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140462 v187) v2 from mds.? 10.252.58.31:6800/10908 -8339> 2013-04-03 10:45:15.596538 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140462 v187) v2 -- ?+0 0x2f79b00 con 0x2118420 -8338> 2013-04-03 10:45:16.943157 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113949 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 177 v187) v2) to leader v1 ==== 560+0+0 (2455319292 0 0) 0x2a83840 con 0x208a840 -8337> 2013-04-03 10:45:16.943179 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -8336> 2013-04-03 10:45:16.943191 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83dc0 from 10.252.24.88:6789/0 -8335> 2013-04-03 10:45:16.943244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 177 v187) v2 from mds.0 10.252.24.25:6802/3144 -8334> 2013-04-03 10:45:16.943255 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 177 v187) v2 tid 17719) v2 -- ?+0 0x24f8b40 con 0x208a840 -8333> 2013-04-03 10:45:17.380783 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427494 fc 427469 pn 0 opn 0) v3 -- ?+0 0x20f4500 -8332> 2013-04-03 10:45:17.381619 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113950 ==== paxos(lease_ack lc 427494 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (2156259249 0 0) 0x20f4500 con 0x208a840 -8331> 2013-04-03 10:45:17.381666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos -8330> 2013-04-03 10:45:17.381686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394100 -8329> 2013-04-03 10:45:17.381703 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8328> 2013-04-03 10:45:17.394486 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 178 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21833 v187) v2 ==== 289+0+0 (1853756841 0 0) 0x2f79000 con 0x200d420 -8327> 2013-04-03 10:45:17.394531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21833 v187) v2 from mds.? 10.252.24.88:6800/17213 -8326> 2013-04-03 10:45:17.394540 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21833 v187) v2 -- ?+0 0x2a83840 con 0x200d420 -8325> 2013-04-03 10:45:18.274606 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113951 ==== forward(pg_stats(1 pgs tid 36 v 605) v1) to leader v1 ==== 757+0+0 (3600122910 0 0) 0x20e3dc0 con 0x208a840 -8324> 2013-04-03 10:45:18.274636 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -8323> 2013-04-03 10:45:18.274646 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -8322> 2013-04-03 10:45:18.274698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 preprocess_query pg_stats(1 pgs tid 36 v 605) v1 from osd.2 10.252.24.30:6801/5656 -8321> 2013-04-03 10:45:18.274722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 prepare_update pg_stats(1 pgs tid 36 v 605) v1 from osd.2 10.252.24.30:6801/5656 -8320> 2013-04-03 10:45:18.274728 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 prepare_pg_stats pg_stats(1 pgs tid 36 v 605) v1 from osd.2 -8319> 2013-04-03 10:45:18.274734 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29106 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1176 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -8318> 2013-04-03 10:45:18.324918 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29106 encode_pending v 29107 -8317> 2013-04-03 10:45:18.324948 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427494) queue_proposal bl 654 bytes; ctx = 0x2c3dc40 -8316> 2013-04-03 10:45:18.324957 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427494) propose_queued 427495 654 bytes -8315> 2013-04-03 10:45:18.326021 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427494 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4780 -8314> 2013-04-03 10:45:18.354175 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113952 ==== paxos(accept lc 427494 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (951663813 0 0) 0x20f4780 con 0x208a840 -8313> 2013-04-03 10:45:18.354581 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427495 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4500 -8312> 2013-04-03 10:45:18.354604 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427495 fc 427469 pn 0 opn 0) v3 -- ?+0 0x29b4c80 -8311> 2013-04-03 10:45:18.354685 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29106 update_from_paxos applying incremental 29107 -8310> 2013-04-03 10:45:18.354706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 v29107: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 61891B/s wr, 0op/s -8309> 2013-04-03 10:45:18.354725 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 send_pg_creates to 0 pgs -8308> 2013-04-03 10:45:18.354729 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 update_logger -8307> 2013-04-03 10:45:18.354740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 create_pending v 29108 -8306> 2013-04-03 10:45:18.354749 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29107 _updated_stats for osd.2 10.252.24.30:6801/5656 -8305> 2013-04-03 10:45:18.354754 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 36) v1 tid 17720) v2 -- ?+0 0x2b26b40 con 0x208a840 -8304> 2013-04-03 10:45:18.354765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 check_osd_map already seen 605 >= 605 -8303> 2013-04-03 10:45:18.354767 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 update_logger -8302> 2013-04-03 10:45:18.354910 7fcdffcdb700 0 log [INF] : pgmap v29107: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 61891B/s wr, 0op/s -8301> 2013-04-03 10:45:18.354917 7fcdffcdb700 10 send_log to self -8300> 2013-04-03 10:45:18.354917 7fcdffcdb700 10 log_queue is 1 last_log 173 sent 172 num 1 unsent 1 sending 1 -8299> 2013-04-03 10:45:18.354919 7fcdffcdb700 10 will send 2013-04-03 10:45:18.354916 mon.0 10.252.24.25:6789/0 173 : [INF] pgmap v29107: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 61891B/s wr, 0op/s -8298> 2013-04-03 10:45:18.354927 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74a00 -8297> 2013-04-03 10:45:18.354983 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos -8296> 2013-04-03 10:45:18.355003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394100 -8295> 2013-04-03 10:45:18.355021 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8294> 2013-04-03 10:45:18.355042 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74a00 con 0x1bac160 -8293> 2013-04-03 10:45:18.355064 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos -8292> 2013-04-03 10:45:18.355079 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394100 -8291> 2013-04-03 10:45:18.355089 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8290> 2013-04-03 10:45:18.355102 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 preprocess_log log(1 entries) v1 from mon.0 -8289> 2013-04-03 10:45:18.355126 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8288> 2013-04-03 10:45:18.355138 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 prepare_log log(1 entries) v1 from mon.0 -8287> 2013-04-03 10:45:18.355148 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 logging 2013-04-03 10:45:18.354916 mon.0 10.252.24.25:6789/0 173 : [INF] pgmap v29107: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 61891B/s wr, 0op/s -8286> 2013-04-03 10:45:18.367335 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113953 ==== paxos(lease_ack lc 427495 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3635712195 0 0) 0x29b4c80 con 0x208a840 -8285> 2013-04-03 10:45:18.367379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos -8284> 2013-04-03 10:45:18.367396 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394100 -8283> 2013-04-03 10:45:18.367411 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8282> 2013-04-03 10:45:18.373694 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29107 check_down_pgs -8281> 2013-04-03 10:45:18.373724 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29107 v29107: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 61891B/s wr, 0op/s -8280> 2013-04-03 10:45:18.373754 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -8279> 2013-04-03 10:45:18.373774 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -8278> 2013-04-03 10:45:18.373811 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -8277> 2013-04-03 10:45:18.373841 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos -8276> 2013-04-03 10:45:18.373858 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394100 update_from_paxos version 394100 summary v 394100 -8275> 2013-04-03 10:45:18.373867 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394100 log -8274> 2013-04-03 10:45:18.373883 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8273> 2013-04-03 10:45:18.373900 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -8272> 2013-04-03 10:45:19.354387 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394100 encode_pending v394101 -8271> 2013-04-03 10:45:19.354413 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427495) queue_proposal bl 444 bytes; ctx = 0x2c3dc80 -8270> 2013-04-03 10:45:19.354421 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427495) propose_queued 427496 444 bytes -8269> 2013-04-03 10:45:19.355836 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427495 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4280 -8268> 2013-04-03 10:45:19.474849 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113954 ==== paxos(accept lc 427495 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3650028504 0 0) 0x20f4280 con 0x208a840 -8267> 2013-04-03 10:45:19.475169 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427496 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4c80 -8266> 2013-04-03 10:45:19.475203 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427496 fc 427469 pn 0 opn 0) v3 -- ?+0 0x20f4780 -8265> 2013-04-03 10:45:19.475242 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos -8264> 2013-04-03 10:45:19.475269 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos version 394101 summary v 394100 -8263> 2013-04-03 10:45:19.475285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos latest full 393798 -8262> 2013-04-03 10:45:19.475304 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394101 update_from_paxos applying incremental log 394101 2013-04-03 10:45:18.354916 mon.0 10.252.24.25:6789/0 173 : [INF] pgmap v29107: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 61891B/s wr, 0op/s -8261> 2013-04-03 10:45:19.475355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 check_subs -8260> 2013-04-03 10:45:19.475387 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 create_pending v 394102 -8259> 2013-04-03 10:45:19.475409 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394101 _updated_log for mon.0 10.252.24.25:6789/0 -8258> 2013-04-03 10:45:19.475422 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 173) v1 -- ?+0 0x2224540 con 0x1bac160 -8257> 2013-04-03 10:45:19.475475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos -8256> 2013-04-03 10:45:19.475491 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos version 394101 summary v 394101 -8255> 2013-04-03 10:45:19.475509 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8254> 2013-04-03 10:45:19.475532 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 173) v1 ==== 0+0+0 (0 0 0) 0x2224540 con 0x1bac160 -8253> 2013-04-03 10:45:19.475540 7fcdffcdb700 10 handle_log_ack log(last 173) v1 -8252> 2013-04-03 10:45:19.475542 7fcdffcdb700 10 logged 2013-04-03 10:45:18.354916 mon.0 10.252.24.25:6789/0 173 : [INF] pgmap v29107: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 61891B/s wr, 0op/s -8251> 2013-04-03 10:45:19.486740 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113955 ==== paxos(lease_ack lc 427496 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (988926813 0 0) 0x20f4780 con 0x208a840 -8250> 2013-04-03 10:45:19.486776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos -8249> 2013-04-03 10:45:19.486792 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos version 394101 summary v 394101 -8248> 2013-04-03 10:45:19.486807 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8247> 2013-04-03 10:45:19.596590 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 179 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140463 v187) v2 ==== 289+0+0 (874476053 0 0) 0x20e4600 con 0x2118420 -8246> 2013-04-03 10:45:19.596666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140463 v187) v2 from mds.? 10.252.58.31:6800/10908 -8245> 2013-04-03 10:45:19.596679 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140463 v187) v2 -- ?+0 0x20e3dc0 con 0x2118420 -8244> 2013-04-03 10:45:20.943260 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113956 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 178 v187) v2) to leader v1 ==== 560+0+0 (2605772672 0 0) 0x2a83b00 con 0x208a840 -8243> 2013-04-03 10:45:20.943281 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -8242> 2013-04-03 10:45:20.943291 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1bae8c0 from 10.252.24.88:6789/0 -8241> 2013-04-03 10:45:20.943344 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 178 v187) v2 from mds.0 10.252.24.25:6802/3144 -8240> 2013-04-03 10:45:20.943356 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 178 v187) v2 tid 17721) v2 -- ?+0 0x24f8b40 con 0x208a840 -8239> 2013-04-03 10:45:21.145214 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3003000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33166/0 -8238> 2013-04-03 10:45:21.145238 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33166/0) -8237> 2013-04-03 10:45:21.145253 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003000 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -8236> 2013-04-03 10:45:21.145474 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3003000 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -8235> 2013-04-03 10:45:21.394620 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 179 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21834 v187) v2 ==== 289+0+0 (198038574 0 0) 0x2f79840 con 0x200d420 -8234> 2013-04-03 10:45:21.394668 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21834 v187) v2 from mds.? 10.252.24.88:6800/17213 -8233> 2013-04-03 10:45:21.394678 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21834 v187) v2 -- ?+0 0x2a83b00 con 0x200d420 -8232> 2013-04-03 10:45:21.628833 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113957 ==== forward(pg_stats(1 pgs tid 54 v 605) v1) to leader v1 ==== 753+0+0 (3182462769 0 0) 0x20e4600 con 0x208a840 -8231> 2013-04-03 10:45:21.628853 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -8230> 2013-04-03 10:45:21.628862 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -8229> 2013-04-03 10:45:21.628914 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 preprocess_query pg_stats(1 pgs tid 54 v 605) v1 from osd.1 10.252.24.25:6805/3572 -8228> 2013-04-03 10:45:21.628939 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 prepare_update pg_stats(1 pgs tid 54 v 605) v1 from osd.1 10.252.24.25:6805/3572 -8227> 2013-04-03 10:45:21.628945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 prepare_pg_stats pg_stats(1 pgs tid 54 v 605) v1 from osd.1 -8226> 2013-04-03 10:45:21.628951 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29107 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -8225> 2013-04-03 10:45:21.679137 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29107 encode_pending v 29108 -8224> 2013-04-03 10:45:21.679171 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427496) queue_proposal bl 650 bytes; ctx = 0x2c3ddb0 -8223> 2013-04-03 10:45:21.679181 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427496) propose_queued 427497 650 bytes -8222> 2013-04-03 10:45:21.680551 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427496 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116a00 -8221> 2013-04-03 10:45:21.712231 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113958 ==== paxos(accept lc 427496 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1425689363 0 0) 0x2116a00 con 0x208a840 -8220> 2013-04-03 10:45:21.712571 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427497 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4780 -8219> 2013-04-03 10:45:21.712599 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427497 fc 427469 pn 0 opn 0) v3 -- ?+0 0x20f4280 -8218> 2013-04-03 10:45:21.712670 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29107 update_from_paxos applying incremental 29108 -8217> 2013-04-03 10:45:21.712686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 v29108: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -8216> 2013-04-03 10:45:21.712706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 send_pg_creates to 0 pgs -8215> 2013-04-03 10:45:21.712712 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 update_logger -8214> 2013-04-03 10:45:21.712723 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 create_pending v 29109 -8213> 2013-04-03 10:45:21.712732 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29108 _updated_stats for osd.1 10.252.24.25:6805/3572 -8212> 2013-04-03 10:45:21.712738 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 54) v1 tid 17722) v2 -- ?+0 0x2b26b40 con 0x208a840 -8211> 2013-04-03 10:45:21.712748 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 check_osd_map already seen 605 >= 605 -8210> 2013-04-03 10:45:21.712751 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 update_logger -8209> 2013-04-03 10:45:21.712761 7fcdffcdb700 0 log [INF] : pgmap v29108: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -8208> 2013-04-03 10:45:21.712763 7fcdffcdb700 10 send_log to self -8207> 2013-04-03 10:45:21.712763 7fcdffcdb700 10 log_queue is 1 last_log 174 sent 173 num 1 unsent 1 sending 1 -8206> 2013-04-03 10:45:21.712765 7fcdffcdb700 10 will send 2013-04-03 10:45:21.712762 mon.0 10.252.24.25:6789/0 174 : [INF] pgmap v29108: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -8205> 2013-04-03 10:45:21.712772 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74a00 -8204> 2013-04-03 10:45:21.712819 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos -8203> 2013-04-03 10:45:21.712839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos version 394101 summary v 394101 -8202> 2013-04-03 10:45:21.712858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8201> 2013-04-03 10:45:21.712878 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74a00 con 0x1bac160 -8200> 2013-04-03 10:45:21.712900 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos -8199> 2013-04-03 10:45:21.712915 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos version 394101 summary v 394101 -8198> 2013-04-03 10:45:21.712924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8197> 2013-04-03 10:45:21.712936 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 preprocess_log log(1 entries) v1 from mon.0 -8196> 2013-04-03 10:45:21.712962 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8195> 2013-04-03 10:45:21.712975 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 prepare_log log(1 entries) v1 from mon.0 -8194> 2013-04-03 10:45:21.712986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 logging 2013-04-03 10:45:21.712762 mon.0 10.252.24.25:6789/0 174 : [INF] pgmap v29108: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -8193> 2013-04-03 10:45:21.724695 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113959 ==== paxos(lease_ack lc 427497 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (1183837716 0 0) 0x20f4280 con 0x208a840 -8192> 2013-04-03 10:45:21.724742 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos -8191> 2013-04-03 10:45:21.724765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394101 update_from_paxos version 394101 summary v 394101 -8190> 2013-04-03 10:45:21.724781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8189> 2013-04-03 10:45:21.795074 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113960 ==== forward(pg_stats(1 pgs tid 30459 v 605) v1) to leader v1 ==== 757+0+0 (831822339 0 0) 0x1bae600 con 0x208a840 -8188> 2013-04-03 10:45:21.795086 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -8187> 2013-04-03 10:45:21.795094 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 -8186> 2013-04-03 10:45:21.795123 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 preprocess_query pg_stats(1 pgs tid 30459 v 605) v1 from osd.3 10.252.24.88:6801/17538 -8185> 2013-04-03 10:45:21.795144 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 prepare_update pg_stats(1 pgs tid 30459 v 605) v1 from osd.3 10.252.24.88:6801/17538 -8184> 2013-04-03 10:45:21.795149 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 prepare_pg_stats pg_stats(1 pgs tid 30459 v 605) v1 from osd.3 -8183> 2013-04-03 10:45:21.795154 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29108 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -8182> 2013-04-03 10:45:22.712428 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29108 encode_pending v 29109 -8181> 2013-04-03 10:45:22.712464 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427497) queue_proposal bl 654 bytes; ctx = 0x2c3da80 -8180> 2013-04-03 10:45:22.712471 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427497) propose_queued 427498 654 bytes -8179> 2013-04-03 10:45:22.713590 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427497 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116000 -8178> 2013-04-03 10:45:22.713943 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394101 encode_pending v394102 -8177> 2013-04-03 10:45:22.713973 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427469..427497) queue_proposal bl 442 bytes; ctx = 0x2c3d9a0 -8176> 2013-04-03 10:45:22.713980 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427469..427497) propose_new_value not active; proposal queued -8175> 2013-04-03 10:45:22.742463 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113961 ==== paxos(accept lc 427497 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (745584752 0 0) 0x2116000 con 0x208a840 -8174> 2013-04-03 10:45:22.743518 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427498 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4280 -8173> 2013-04-03 10:45:22.743546 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427498 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2116a00 -8172> 2013-04-03 10:45:22.743618 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29108 update_from_paxos applying incremental 29109 -8171> 2013-04-03 10:45:22.743643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29109 v29109: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1166B/s wr, 0op/s -8170> 2013-04-03 10:45:22.743665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29109 send_pg_creates to 0 pgs -8169> 2013-04-03 10:45:22.743669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29109 update_logger -8168> 2013-04-03 10:45:22.743679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29109 create_pending v 29110 -8167> 2013-04-03 10:45:22.743690 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29109 _updated_stats for osd.3 10.252.24.88:6801/17538 -8166> 2013-04-03 10:45:22.743698 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30459) v1 tid 17723) v2 -- ?+0 0x24f8b40 con 0x208a840 -8165> 2013-04-03 10:45:22.743875 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29109 check_osd_map already seen 605 >= 605 -8164> 2013-04-03 10:45:22.743881 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29109 update_logger -8163> 2013-04-03 10:45:22.743892 7fcdffcdb700 0 log [INF] : pgmap v29109: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1166B/s wr, 0op/s -8162> 2013-04-03 10:45:22.743893 7fcdffcdb700 10 send_log to self -8161> 2013-04-03 10:45:22.743894 7fcdffcdb700 10 log_queue is 2 last_log 175 sent 174 num 2 unsent 1 sending 1 -8160> 2013-04-03 10:45:22.743897 7fcdffcdb700 10 will send 2013-04-03 10:45:21.712762 mon.0 10.252.24.25:6789/0 174 : [INF] pgmap v29108: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -8159> 2013-04-03 10:45:22.743904 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75800 -8158> 2013-04-03 10:45:22.743930 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427498) propose_queued 427499 442 bytes -8157> 2013-04-03 10:45:22.744167 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427498 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4a00 -8156> 2013-04-03 10:45:22.744207 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75800 con 0x1bac160 -8155> 2013-04-03 10:45:22.756194 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113962 ==== paxos(lease_ack lc 427498 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (31880046 0 0) 0x20f4a00 con 0x208a840 -8154> 2013-04-03 10:45:22.768449 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113963 ==== paxos(accept lc 427498 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1259764925 0 0) 0x2116a00 con 0x208a840 -8153> 2013-04-03 10:45:22.768761 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427499 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4a00 -8152> 2013-04-03 10:45:22.768793 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427499 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2116000 -8151> 2013-04-03 10:45:22.768826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos -8150> 2013-04-03 10:45:22.768851 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394101 -8149> 2013-04-03 10:45:22.768865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos latest full 393798 -8148> 2013-04-03 10:45:22.768883 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394102 update_from_paxos applying incremental log 394102 2013-04-03 10:45:21.712762 mon.0 10.252.24.25:6789/0 174 : [INF] pgmap v29108: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -8147> 2013-04-03 10:45:22.768925 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 check_subs -8146> 2013-04-03 10:45:22.768956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 create_pending v 394103 -8145> 2013-04-03 10:45:22.768978 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394102 _updated_log for mon.0 10.252.24.25:6789/0 -8144> 2013-04-03 10:45:22.768990 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 174) v1 -- ?+0 0x2224e00 con 0x1bac160 -8143> 2013-04-03 10:45:22.769010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos -8142> 2013-04-03 10:45:22.769025 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394102 -8141> 2013-04-03 10:45:22.769034 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -8140> 2013-04-03 10:45:22.769045 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 preprocess_log log(1 entries) v1 from mon.0 -8139> 2013-04-03 10:45:22.769063 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 nothing new -8138> 2013-04-03 10:45:22.769113 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos -8137> 2013-04-03 10:45:22.769128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394102 -8136> 2013-04-03 10:45:22.769146 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8135> 2013-04-03 10:45:22.769167 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 174) v1 ==== 0+0+0 (0 0 0) 0x2224e00 con 0x1bac160 -8134> 2013-04-03 10:45:22.769175 7fcdffcdb700 10 handle_log_ack log(last 174) v1 -8133> 2013-04-03 10:45:22.769176 7fcdffcdb700 10 logged 2013-04-03 10:45:21.712762 mon.0 10.252.24.25:6789/0 174 : [INF] pgmap v29108: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -8132> 2013-04-03 10:45:22.782242 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113964 ==== paxos(lease_ack lc 427499 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (963682424 0 0) 0x2116000 con 0x208a840 -8131> 2013-04-03 10:45:22.782278 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos -8130> 2013-04-03 10:45:22.782294 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394102 -8129> 2013-04-03 10:45:22.782310 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8128> 2013-04-03 10:45:23.374041 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29109 check_down_pgs -8127> 2013-04-03 10:45:23.374069 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29109 v29109: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1166B/s wr, 0op/s -8126> 2013-04-03 10:45:23.374096 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -8125> 2013-04-03 10:45:23.374120 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -8124> 2013-04-03 10:45:23.374156 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -8123> 2013-04-03 10:45:23.374187 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos -8122> 2013-04-03 10:45:23.374206 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394102 -8121> 2013-04-03 10:45:23.374215 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394102 log -8120> 2013-04-03 10:45:23.374232 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8119> 2013-04-03 10:45:23.374251 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -8118> 2013-04-03 10:45:23.596715 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 180 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140464 v187) v2 ==== 289+0+0 (4290566477 0 0) 0x20e3b00 con 0x2118420 -8117> 2013-04-03 10:45:23.596789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140464 v187) v2 from mds.? 10.252.58.31:6800/10908 -8116> 2013-04-03 10:45:23.596802 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140464 v187) v2 -- ?+0 0x1bae600 con 0x2118420 -8115> 2013-04-03 10:45:24.943408 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113965 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 179 v187) v2) to leader v1 ==== 560+0+0 (2812568506 0 0) 0x1bae340 con 0x208a840 -8114> 2013-04-03 10:45:24.943430 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -8113> 2013-04-03 10:45:24.943440 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2efa340 from 10.252.24.88:6789/0 -8112> 2013-04-03 10:45:24.943992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 179 v187) v2 from mds.0 10.252.24.25:6802/3144 -8111> 2013-04-03 10:45:24.944009 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 179 v187) v2 tid 17724) v2 -- ?+0 0x2b26b40 con 0x208a840 -8110> 2013-04-03 10:45:25.394748 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 180 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21835 v187) v2 ==== 289+0+0 (681798483 0 0) 0x2ce1580 con 0x200d420 -8109> 2013-04-03 10:45:25.394820 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21835 v187) v2 from mds.? 10.252.24.88:6800/17213 -8108> 2013-04-03 10:45:25.394832 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21835 v187) v2 -- ?+0 0x1bae340 con 0x200d420 -8107> 2013-04-03 10:45:25.768871 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427499 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2116780 -8106> 2013-04-03 10:45:25.769757 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113966 ==== paxos(lease_ack lc 427499 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3417556181 0 0) 0x2116780 con 0x208a840 -8105> 2013-04-03 10:45:25.769823 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos -8104> 2013-04-03 10:45:25.769842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394102 -8103> 2013-04-03 10:45:25.769859 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8102> 2013-04-03 10:45:27.596838 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 181 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140465 v187) v2 ==== 289+0+0 (3704776240 0 0) 0x20e3840 con 0x2118420 -8101> 2013-04-03 10:45:27.596913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140465 v187) v2 from mds.? 10.252.58.31:6800/10908 -8100> 2013-04-03 10:45:27.596926 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140465 v187) v2 -- ?+0 0x2ce1580 con 0x2118420 -8099> 2013-04-03 10:45:28.374398 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29109 v29109: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1166B/s wr, 0op/s -8098> 2013-04-03 10:45:28.374443 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -8097> 2013-04-03 10:45:28.374466 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -8096> 2013-04-03 10:45:28.374508 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -8095> 2013-04-03 10:45:28.374539 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos -8094> 2013-04-03 10:45:28.374557 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394102 -8093> 2013-04-03 10:45:28.374916 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394102 log -8092> 2013-04-03 10:45:28.374941 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8091> 2013-04-03 10:45:28.374960 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -8090> 2013-04-03 10:45:28.768949 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427499 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3062280 -8089> 2013-04-03 10:45:28.769837 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113967 ==== paxos(lease_ack lc 427499 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3250285382 0 0) 0x3062280 con 0x208a840 -8088> 2013-04-03 10:45:28.769906 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos -8087> 2013-04-03 10:45:28.769925 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394102 update_from_paxos version 394102 summary v 394102 -8086> 2013-04-03 10:45:28.769943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -8085> 2013-04-03 10:45:28.871382 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113968 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (424405211 0 0) 0x20e3b00 con 0x208a840 -8084> 2013-04-03 10:45:28.871396 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -8083> 2013-04-03 10:45:28.871401 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -8082> 2013-04-03 10:45:28.871408 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 -8081> 2013-04-03 10:45:28.871411 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( start_reply flags( reply-to(mon.2 10.252.24.88:6789/0) ) ) v1 -- ?+0 0x20e3840 -8080> 2013-04-03 10:45:28.943591 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113969 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 180 v187) v2) to leader v1 ==== 560+0+0 (885832174 0 0) 0x2efa600 con 0x208a840 -8079> 2013-04-03 10:45:28.943605 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -8078> 2013-04-03 10:45:28.943614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ef9b00 from 10.252.24.88:6789/0 -8077> 2013-04-03 10:45:28.943647 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 180 v187) v2 from mds.0 10.252.24.25:6802/3144 -8076> 2013-04-03 10:45:28.943659 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 180 v187) v2 tid 17725) v2 -- ?+0 0x24f8b40 con 0x208a840 -8075> 2013-04-03 10:45:29.394849 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 181 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21836 v187) v2 ==== 289+0+0 (3232388896 0 0) 0x2ce2340 con 0x200d420 -8074> 2013-04-03 10:45:29.394921 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21836 v187) v2 from mds.? 10.252.24.88:6800/17213 -8073> 2013-04-03 10:45:29.394937 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21836 v187) v2 -- ?+0 0x2efa600 con 0x200d420 -8072> 2013-04-03 10:45:31.596977 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 182 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140466 v187) v2 ==== 289+0+0 (3110184887 0 0) 0x20e4340 con 0x2118420 -8071> 2013-04-03 10:45:31.597054 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140466 v187) v2 from mds.? 10.252.58.31:6800/10908 -8070> 2013-04-03 10:45:31.597073 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140466 v187) v2 -- ?+0 0x2ce2340 con 0x2118420 -8069> 2013-04-03 10:45:31.630165 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113970 ==== forward(pg_stats(1 pgs tid 55 v 605) v1) to leader v1 ==== 753+0+0 (1293572641 0 0) 0x20e3b00 con 0x208a840 -8068> 2013-04-03 10:45:31.630179 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -8067> 2013-04-03 10:45:31.630188 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -8066> 2013-04-03 10:45:31.630221 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29109 preprocess_query pg_stats(1 pgs tid 55 v 605) v1 from osd.1 10.252.24.25:6805/3572 -8065> 2013-04-03 10:45:31.630246 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29109 prepare_update pg_stats(1 pgs tid 55 v 605) v1 from osd.1 10.252.24.25:6805/3572 -8064> 2013-04-03 10:45:31.630252 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29109 prepare_pg_stats pg_stats(1 pgs tid 55 v 605) v1 from osd.1 -8063> 2013-04-03 10:45:31.630259 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29109 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -8062> 2013-04-03 10:45:31.680445 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29109 encode_pending v 29110 -8061> 2013-04-03 10:45:31.680484 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427499) queue_proposal bl 650 bytes; ctx = 0x2c3d980 -8060> 2013-04-03 10:45:31.680494 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427499) propose_queued 427500 650 bytes -8059> 2013-04-03 10:45:31.681602 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427499 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003280 -8058> 2013-04-03 10:45:31.887085 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113971 ==== paxos(accept lc 427499 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3330228152 0 0) 0x3003280 con 0x208a840 -8057> 2013-04-03 10:45:31.887402 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427500 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062280 -8056> 2013-04-03 10:45:31.887431 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427500 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2116780 -8055> 2013-04-03 10:45:31.887497 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29109 update_from_paxos applying incremental 29110 -8054> 2013-04-03 10:45:31.887514 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 v29110: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 511B/s wr, 0op/s -8053> 2013-04-03 10:45:31.887534 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 send_pg_creates to 0 pgs -8052> 2013-04-03 10:45:31.887538 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 update_logger -8051> 2013-04-03 10:45:31.887549 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 create_pending v 29111 -8050> 2013-04-03 10:45:31.887563 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 _updated_stats for osd.1 10.252.24.25:6805/3572 -8049> 2013-04-03 10:45:31.887570 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 55) v1 tid 17726) v2 -- ?+0 0x2b26b40 con 0x208a840 -8048> 2013-04-03 10:45:31.887579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 check_osd_map already seen 605 >= 605 -8047> 2013-04-03 10:45:31.887583 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 update_logger -8046> 2013-04-03 10:45:31.887593 7fcdffcdb700 0 log [INF] : pgmap v29110: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 511B/s wr, 0op/s -8045> 2013-04-03 10:45:31.887595 7fcdffcdb700 10 send_log to self -8044> 2013-04-03 10:45:31.887596 7fcdffcdb700 10 log_queue is 2 last_log 176 sent 174 num 2 unsent 2 sending 2 -8043> 2013-04-03 10:45:31.887598 7fcdffcdb700 10 will send 2013-04-03 10:45:22.743893 mon.0 10.252.24.25:6789/0 175 : [INF] pgmap v29109: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1166B/s wr, 0op/s -8042> 2013-04-03 10:45:31.887603 7fcdffcdb700 10 will send 2013-04-03 10:45:31.887595 mon.0 10.252.24.25:6789/0 176 : [INF] pgmap v29110: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 511B/s wr, 0op/s -8041> 2013-04-03 10:45:31.887609 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b75800 -8040> 2013-04-03 10:45:31.887656 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 update_from_paxos -8039> 2013-04-03 10:45:31.887676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 update_from_paxos version 394102 summary v 394102 -8038> 2013-04-03 10:45:31.887695 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8037> 2013-04-03 10:45:31.887717 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75800 con 0x1bac160 -8036> 2013-04-03 10:45:31.887739 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 update_from_paxos -8035> 2013-04-03 10:45:31.887754 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 update_from_paxos version 394102 summary v 394102 -8034> 2013-04-03 10:45:31.887764 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -8033> 2013-04-03 10:45:31.887777 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 preprocess_log log(2 entries) v1 from mon.0 -8032> 2013-04-03 10:45:31.887809 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -8031> 2013-04-03 10:45:31.887825 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 prepare_log log(2 entries) v1 from mon.0 -8030> 2013-04-03 10:45:31.887836 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 logging 2013-04-03 10:45:22.743893 mon.0 10.252.24.25:6789/0 175 : [INF] pgmap v29109: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1166B/s wr, 0op/s -8029> 2013-04-03 10:45:31.887860 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 logging 2013-04-03 10:45:31.887595 mon.0 10.252.24.25:6789/0 176 : [INF] pgmap v29110: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 511B/s wr, 0op/s -8028> 2013-04-03 10:45:31.913236 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113972 ==== paxos(lease_ack lc 427500 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3665283951 0 0) 0x2116780 con 0x208a840 -8027> 2013-04-03 10:45:31.913273 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 update_from_paxos -8026> 2013-04-03 10:45:31.913290 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 update_from_paxos version 394102 summary v 394102 -8025> 2013-04-03 10:45:31.913307 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8024> 2013-04-03 10:45:32.887287 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394102 encode_pending v394103 -8023> 2013-04-03 10:45:32.887316 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427500) queue_proposal bl 725 bytes; ctx = 0x2c3dac0 -8022> 2013-04-03 10:45:32.887324 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427500) propose_queued 427501 725 bytes -8021> 2013-04-03 10:45:32.888432 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427500 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003000 -8020> 2013-04-03 10:45:32.919489 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113973 ==== paxos(accept lc 427500 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2344479248 0 0) 0x3003000 con 0x208a840 -8019> 2013-04-03 10:45:32.920344 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427501 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116780 -8018> 2013-04-03 10:45:32.920377 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427501 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3003280 -8017> 2013-04-03 10:45:32.920417 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos -8016> 2013-04-03 10:45:32.920443 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos version 394103 summary v 394102 -8015> 2013-04-03 10:45:32.920457 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos latest full 393798 -8014> 2013-04-03 10:45:32.920476 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos applying incremental log 394103 2013-04-03 10:45:22.743893 mon.0 10.252.24.25:6789/0 175 : [INF] pgmap v29109: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1166B/s wr, 0op/s -8013> 2013-04-03 10:45:32.920507 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos applying incremental log 394103 2013-04-03 10:45:31.887595 mon.0 10.252.24.25:6789/0 176 : [INF] pgmap v29110: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 511B/s wr, 0op/s -8012> 2013-04-03 10:45:32.920548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 check_subs -8011> 2013-04-03 10:45:32.920580 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 create_pending v 394104 -8010> 2013-04-03 10:45:32.920603 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 _updated_log for mon.0 10.252.24.25:6789/0 -8009> 2013-04-03 10:45:32.920616 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 176) v1 -- ?+0 0x2224700 con 0x1bac160 -8008> 2013-04-03 10:45:32.920670 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos -8007> 2013-04-03 10:45:32.920686 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos version 394103 summary v 394103 -8006> 2013-04-03 10:45:32.920705 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -8005> 2013-04-03 10:45:32.920727 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 176) v1 ==== 0+0+0 (0 0 0) 0x2224700 con 0x1bac160 -8004> 2013-04-03 10:45:32.920735 7fcdffcdb700 10 handle_log_ack log(last 176) v1 -8003> 2013-04-03 10:45:32.920737 7fcdffcdb700 10 logged 2013-04-03 10:45:22.743893 mon.0 10.252.24.25:6789/0 175 : [INF] pgmap v29109: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1166B/s wr, 0op/s -8002> 2013-04-03 10:45:32.920743 7fcdffcdb700 10 logged 2013-04-03 10:45:31.887595 mon.0 10.252.24.25:6789/0 176 : [INF] pgmap v29110: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 511B/s wr, 0op/s -8001> 2013-04-03 10:45:32.933081 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113974 ==== paxos(lease_ack lc 427501 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3618994348 0 0) 0x3003280 con 0x208a840 -8000> 2013-04-03 10:45:32.933116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos -7999> 2013-04-03 10:45:32.933133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos version 394103 summary v 394103 -7998> 2013-04-03 10:45:32.933150 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7997> 2013-04-03 10:45:32.943527 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113975 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 181 v187) v2) to leader v1 ==== 560+0+0 (3921263803 0 0) 0x2145580 con 0x208a840 -7996> 2013-04-03 10:45:32.943539 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -7995> 2013-04-03 10:45:32.943548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2145b00 from 10.252.24.88:6789/0 -7994> 2013-04-03 10:45:32.943577 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 181 v187) v2 from mds.0 10.252.24.25:6802/3144 -7993> 2013-04-03 10:45:32.943588 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 181 v187) v2 tid 17727) v2 -- ?+0 0x24f8000 con 0x208a840 -7992> 2013-04-03 10:45:33.276576 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113976 ==== forward(pg_stats(1 pgs tid 37 v 605) v1) to leader v1 ==== 757+0+0 (2977222140 0 0) 0x20e3b00 con 0x208a840 -7991> 2013-04-03 10:45:33.276603 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -7990> 2013-04-03 10:45:33.276614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 -7989> 2013-04-03 10:45:33.276667 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 preprocess_query pg_stats(1 pgs tid 37 v 605) v1 from osd.2 10.252.24.30:6801/5656 -7988> 2013-04-03 10:45:33.276692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 prepare_update pg_stats(1 pgs tid 37 v 605) v1 from osd.2 10.252.24.30:6801/5656 -7987> 2013-04-03 10:45:33.276698 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 prepare_pg_stats pg_stats(1 pgs tid 37 v 605) v1 from osd.2 -7986> 2013-04-03 10:45:33.276705 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -7985> 2013-04-03 10:45:33.375074 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 check_down_pgs -7984> 2013-04-03 10:45:33.375103 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 v29110: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 511B/s wr, 0op/s -7983> 2013-04-03 10:45:33.375137 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -7982> 2013-04-03 10:45:33.375160 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -7981> 2013-04-03 10:45:33.375197 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -7980> 2013-04-03 10:45:33.375228 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos -7979> 2013-04-03 10:45:33.375247 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos version 394103 summary v 394103 -7978> 2013-04-03 10:45:33.375257 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 log -7977> 2013-04-03 10:45:33.375274 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7976> 2013-04-03 10:45:33.375293 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -7975> 2013-04-03 10:45:33.394978 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 182 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21837 v187) v2 ==== 289+0+0 (3821332573 0 0) 0x2f79580 con 0x200d420 -7974> 2013-04-03 10:45:33.395024 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21837 v187) v2 from mds.? 10.252.24.88:6800/17213 -7973> 2013-04-03 10:45:33.395035 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21837 v187) v2 -- ?+0 0x20e3b00 con 0x200d420 -7972> 2013-04-03 10:45:33.919695 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 encode_pending v 29111 -7971> 2013-04-03 10:45:33.919738 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427501) queue_proposal bl 654 bytes; ctx = 0x2c3db00 -7970> 2013-04-03 10:45:33.919748 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427501) propose_queued 427502 654 bytes -7969> 2013-04-03 10:45:33.921064 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427501 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3170c80 -7968> 2013-04-03 10:45:33.944879 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113977 ==== paxos(accept lc 427501 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (840582011 0 0) 0x3170c80 con 0x208a840 -7967> 2013-04-03 10:45:33.945212 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427502 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003280 -7966> 2013-04-03 10:45:33.945241 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427502 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3003000 -7965> 2013-04-03 10:45:33.945315 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29110 update_from_paxos applying incremental 29111 -7964> 2013-04-03 10:45:33.945336 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 v29111: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1461B/s wr, 0op/s -7963> 2013-04-03 10:45:33.945356 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 send_pg_creates to 0 pgs -7962> 2013-04-03 10:45:33.945361 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 update_logger -7961> 2013-04-03 10:45:33.945371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 create_pending v 29112 -7960> 2013-04-03 10:45:33.945381 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 _updated_stats for osd.2 10.252.24.30:6801/5656 -7959> 2013-04-03 10:45:33.945389 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 37) v1 tid 17728) v2 -- ?+0 0x2b26b40 con 0x208a840 -7958> 2013-04-03 10:45:33.945399 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 check_osd_map already seen 605 >= 605 -7957> 2013-04-03 10:45:33.945402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 update_logger -7956> 2013-04-03 10:45:33.945417 7fcdffcdb700 0 log [INF] : pgmap v29111: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1461B/s wr, 0op/s -7955> 2013-04-03 10:45:33.945418 7fcdffcdb700 10 send_log to self -7954> 2013-04-03 10:45:33.945419 7fcdffcdb700 10 log_queue is 1 last_log 177 sent 176 num 1 unsent 1 sending 1 -7953> 2013-04-03 10:45:33.945421 7fcdffcdb700 10 will send 2013-04-03 10:45:33.945418 mon.0 10.252.24.25:6789/0 177 : [INF] pgmap v29111: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1461B/s wr, 0op/s -7952> 2013-04-03 10:45:33.945429 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75800 -7951> 2013-04-03 10:45:33.945482 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos -7950> 2013-04-03 10:45:33.945501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos version 394103 summary v 394103 -7949> 2013-04-03 10:45:33.945521 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7948> 2013-04-03 10:45:33.945544 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75800 con 0x1bac160 -7947> 2013-04-03 10:45:33.945566 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos -7946> 2013-04-03 10:45:33.945582 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos version 394103 summary v 394103 -7945> 2013-04-03 10:45:33.945591 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7944> 2013-04-03 10:45:33.945603 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 preprocess_log log(1 entries) v1 from mon.0 -7943> 2013-04-03 10:45:33.945628 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7942> 2013-04-03 10:45:33.945640 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 prepare_log log(1 entries) v1 from mon.0 -7941> 2013-04-03 10:45:33.945651 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 logging 2013-04-03 10:45:33.945418 mon.0 10.252.24.25:6789/0 177 : [INF] pgmap v29111: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1461B/s wr, 0op/s -7940> 2013-04-03 10:45:33.970825 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113978 ==== paxos(lease_ack lc 427502 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3219970420 0 0) 0x3003000 con 0x208a840 -7939> 2013-04-03 10:45:33.970862 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos -7938> 2013-04-03 10:45:33.970879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 update_from_paxos version 394103 summary v 394103 -7937> 2013-04-03 10:45:33.970897 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7936> 2013-04-03 10:45:34.945072 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394103 encode_pending v394104 -7935> 2013-04-03 10:45:34.945103 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427502) queue_proposal bl 443 bytes; ctx = 0x2c3dda0 -7934> 2013-04-03 10:45:34.945111 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427502) propose_queued 427503 443 bytes -7933> 2013-04-03 10:45:34.946164 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427502 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3170a00 -7932> 2013-04-03 10:45:34.995882 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113979 ==== paxos(accept lc 427502 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1798099078 0 0) 0x3170a00 con 0x208a840 -7931> 2013-04-03 10:45:34.996173 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427503 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003000 -7930> 2013-04-03 10:45:34.996202 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427503 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3170c80 -7929> 2013-04-03 10:45:34.996238 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos -7928> 2013-04-03 10:45:34.996265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos version 394104 summary v 394103 -7927> 2013-04-03 10:45:34.996281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos latest full 393798 -7926> 2013-04-03 10:45:34.996301 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos applying incremental log 394104 2013-04-03 10:45:33.945418 mon.0 10.252.24.25:6789/0 177 : [INF] pgmap v29111: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1461B/s wr, 0op/s -7925> 2013-04-03 10:45:34.996349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 check_subs -7924> 2013-04-03 10:45:34.996382 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 create_pending v 394105 -7923> 2013-04-03 10:45:34.996405 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 _updated_log for mon.0 10.252.24.25:6789/0 -7922> 2013-04-03 10:45:34.996418 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 177) v1 -- ?+0 0x2224380 con 0x1bac160 -7921> 2013-04-03 10:45:34.996473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos -7920> 2013-04-03 10:45:34.996490 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos version 394104 summary v 394104 -7919> 2013-04-03 10:45:34.996509 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7918> 2013-04-03 10:45:34.996532 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 177) v1 ==== 0+0+0 (0 0 0) 0x2224380 con 0x1bac160 -7917> 2013-04-03 10:45:34.996541 7fcdffcdb700 10 handle_log_ack log(last 177) v1 -7916> 2013-04-03 10:45:34.996543 7fcdffcdb700 10 logged 2013-04-03 10:45:33.945418 mon.0 10.252.24.25:6789/0 177 : [INF] pgmap v29111: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 1461B/s wr, 0op/s -7915> 2013-04-03 10:45:35.008209 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113980 ==== paxos(lease_ack lc 427503 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (2730955605 0 0) 0x3170c80 con 0x208a840 -7914> 2013-04-03 10:45:35.008247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos -7913> 2013-04-03 10:45:35.008264 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos version 394104 summary v 394104 -7912> 2013-04-03 10:45:35.008281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7911> 2013-04-03 10:45:35.530784 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113981 ==== forward(pg_stats(0 pgs tid 63 v 605) v1) to leader v1 ==== 368+0+0 (1009100059 0 0) 0x2146080 con 0x208a840 -7910> 2013-04-03 10:45:35.530810 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 -7909> 2013-04-03 10:45:35.530821 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 -7908> 2013-04-03 10:45:35.530874 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 preprocess_query pg_stats(0 pgs tid 63 v 605) v1 from osd.4 10.252.24.32:6800/13473 -7907> 2013-04-03 10:45:35.530899 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 prepare_update pg_stats(0 pgs tid 63 v 605) v1 from osd.4 10.252.24.32:6800/13473 -7906> 2013-04-03 10:45:35.530906 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 prepare_pg_stats pg_stats(0 pgs tid 63 v 605) v1 from osd.4 -7905> 2013-04-03 10:45:35.530913 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 got osd.4 osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[])) -7904> 2013-04-03 10:45:35.597091 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 183 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140467 v187) v2 ==== 289+0+0 (2584680650 0 0) 0x2cc58c0 con 0x2118420 -7903> 2013-04-03 10:45:35.597150 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140467 v187) v2 from mds.? 10.252.58.31:6800/10908 -7902> 2013-04-03 10:45:35.597162 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140467 v187) v2 -- ?+0 0x2146080 con 0x2118420 -7901> 2013-04-03 10:45:35.996082 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 encode_pending v 29112 -7900> 2013-04-03 10:45:35.996122 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427503) queue_proposal bl 265 bytes; ctx = 0x2c3dd70 -7899> 2013-04-03 10:45:35.996135 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427503) propose_queued 427504 265 bytes -7898> 2013-04-03 10:45:35.997369 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427503 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3170780 -7897> 2013-04-03 10:45:36.174908 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113982 ==== paxos(accept lc 427503 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2889290656 0 0) 0x3170780 con 0x208a840 -7896> 2013-04-03 10:45:36.175716 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427504 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3170c80 -7895> 2013-04-03 10:45:36.175746 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427504 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3170a00 -7894> 2013-04-03 10:45:36.175832 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29111 update_from_paxos applying incremental 29112 -7893> 2013-04-03 10:45:36.175845 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 v29112: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 2847B/s wr, 0op/s -7892> 2013-04-03 10:45:36.175863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 send_pg_creates to 0 pgs -7891> 2013-04-03 10:45:36.175870 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 update_logger -7890> 2013-04-03 10:45:36.175880 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 create_pending v 29113 -7889> 2013-04-03 10:45:36.175890 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 _updated_stats for osd.4 10.252.24.32:6800/13473 -7888> 2013-04-03 10:45:36.175898 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(0 pgs tid 63) v1 tid 17729) v2 -- ?+0 0x24f8000 con 0x208a840 -7887> 2013-04-03 10:45:36.175908 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 check_osd_map already seen 605 >= 605 -7886> 2013-04-03 10:45:36.175911 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 update_logger -7885> 2013-04-03 10:45:36.175922 7fcdffcdb700 0 log [INF] : pgmap v29112: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 2847B/s wr, 0op/s -7884> 2013-04-03 10:45:36.175924 7fcdffcdb700 10 send_log to self -7883> 2013-04-03 10:45:36.175924 7fcdffcdb700 10 log_queue is 1 last_log 178 sent 177 num 1 unsent 1 sending 1 -7882> 2013-04-03 10:45:36.175926 7fcdffcdb700 10 will send 2013-04-03 10:45:36.175923 mon.0 10.252.24.25:6789/0 178 : [INF] pgmap v29112: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 2847B/s wr, 0op/s -7881> 2013-04-03 10:45:36.175934 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75800 -7880> 2013-04-03 10:45:36.175985 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos -7879> 2013-04-03 10:45:36.176005 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos version 394104 summary v 394104 -7878> 2013-04-03 10:45:36.176024 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7877> 2013-04-03 10:45:36.176047 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75800 con 0x1bac160 -7876> 2013-04-03 10:45:36.176070 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos -7875> 2013-04-03 10:45:36.176086 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos version 394104 summary v 394104 -7874> 2013-04-03 10:45:36.176095 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7873> 2013-04-03 10:45:36.176107 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 preprocess_log log(1 entries) v1 from mon.0 -7872> 2013-04-03 10:45:36.176136 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7871> 2013-04-03 10:45:36.176149 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 prepare_log log(1 entries) v1 from mon.0 -7870> 2013-04-03 10:45:36.176160 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 logging 2013-04-03 10:45:36.175923 mon.0 10.252.24.25:6789/0 178 : [INF] pgmap v29112: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 2847B/s wr, 0op/s -7869> 2013-04-03 10:45:36.200889 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113983 ==== paxos(lease_ack lc 427504 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (491878946 0 0) 0x2116500 con 0x208a840 -7868> 2013-04-03 10:45:36.200930 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos -7867> 2013-04-03 10:45:36.200947 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 update_from_paxos version 394104 summary v 394104 -7866> 2013-04-03 10:45:36.200962 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7865> 2013-04-03 10:45:36.855561 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113984 ==== forward(pg_stats(0 pgs tid 412 v 605) v1) to leader v1 ==== 384+0+0 (3444263443 0 0) 0x2145840 con 0x208a840 -7864> 2013-04-03 10:45:36.855582 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.0 10.252.58.31:6802/2682 via mon.2 10.252.24.88:6789/0 -7863> 2013-04-03 10:45:36.855593 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 -7862> 2013-04-03 10:45:36.855645 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 preprocess_query pg_stats(0 pgs tid 412 v 605) v1 from osd.0 10.252.58.31:6802/2682 -7861> 2013-04-03 10:45:36.855670 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 prepare_update pg_stats(0 pgs tid 412 v 605) v1 from osd.0 10.252.58.31:6802/2682 -7860> 2013-04-03 10:45:36.855676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 prepare_pg_stats pg_stats(0 pgs tid 412 v 605) v1 from osd.0 -7859> 2013-04-03 10:45:36.855683 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 got osd.0 osd_stat(1141 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1141 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -7858> 2013-04-03 10:45:36.943746 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113985 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 182 v187) v2) to leader v1 ==== 560+0+0 (1401611517 0 0) 0x2145dc0 con 0x208a840 -7857> 2013-04-03 10:45:36.943761 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -7856> 2013-04-03 10:45:36.943769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x21452c0 from 10.252.24.88:6789/0 -7855> 2013-04-03 10:45:36.943804 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 182 v187) v2 from mds.0 10.252.24.25:6802/3144 -7854> 2013-04-03 10:45:36.943816 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 182 v187) v2 tid 17731) v2 -- ?+0 0x2b26b40 con 0x208a840 -7853> 2013-04-03 10:45:37.175122 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 encode_pending v 29113 -7852> 2013-04-03 10:45:37.175153 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427504) queue_proposal bl 281 bytes; ctx = 0x2c3dd90 -7851> 2013-04-03 10:45:37.175161 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427504) propose_queued 427505 281 bytes -7850> 2013-04-03 10:45:37.176291 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427504 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3170500 -7849> 2013-04-03 10:45:37.176404 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394104 encode_pending v394105 -7848> 2013-04-03 10:45:37.176435 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427469..427504) queue_proposal bl 443 bytes; ctx = 0x2c3db60 -7847> 2013-04-03 10:45:37.176443 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427469..427504) propose_new_value not active; proposal queued -7846> 2013-04-03 10:45:37.234283 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113986 ==== paxos(accept lc 427504 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1887186984 0 0) 0x3170500 con 0x208a840 -7845> 2013-04-03 10:45:37.235777 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427505 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116500 -7844> 2013-04-03 10:45:37.235812 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427505 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3170780 -7843> 2013-04-03 10:45:37.235882 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29112 update_from_paxos applying incremental 29113 -7842> 2013-04-03 10:45:37.235895 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 v29113: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail -7841> 2013-04-03 10:45:37.235912 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 send_pg_creates to 0 pgs -7840> 2013-04-03 10:45:37.235916 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 update_logger -7839> 2013-04-03 10:45:37.235927 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 create_pending v 29114 -7838> 2013-04-03 10:45:37.235937 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 _updated_stats for osd.0 10.252.58.31:6802/2682 -7837> 2013-04-03 10:45:37.235944 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(0 pgs tid 412) v1 tid 17730) v2 -- ?+0 0x24f8d80 con 0x208a840 -7836> 2013-04-03 10:45:37.235954 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 check_osd_map already seen 605 >= 605 -7835> 2013-04-03 10:45:37.235958 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 update_logger -7834> 2013-04-03 10:45:37.235968 7fcdffcdb700 0 log [INF] : pgmap v29113: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail -7833> 2013-04-03 10:45:37.235969 7fcdffcdb700 10 send_log to self -7832> 2013-04-03 10:45:37.235970 7fcdffcdb700 10 log_queue is 2 last_log 179 sent 178 num 2 unsent 1 sending 1 -7831> 2013-04-03 10:45:37.235972 7fcdffcdb700 10 will send 2013-04-03 10:45:36.175923 mon.0 10.252.24.25:6789/0 178 : [INF] pgmap v29112: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 2847B/s wr, 0op/s -7830> 2013-04-03 10:45:37.235978 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75c00 -7829> 2013-04-03 10:45:37.236007 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427505) propose_queued 427506 443 bytes -7828> 2013-04-03 10:45:37.236249 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427505 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116000 -7827> 2013-04-03 10:45:37.236290 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75c00 con 0x1bac160 -7826> 2013-04-03 10:45:37.247986 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113987 ==== paxos(lease_ack lc 427505 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (2155355615 0 0) 0x2116000 con 0x208a840 -7825> 2013-04-03 10:45:37.260326 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113988 ==== paxos(accept lc 427505 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2818284249 0 0) 0x3170780 con 0x208a840 -7824> 2013-04-03 10:45:37.260738 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427506 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116000 -7823> 2013-04-03 10:45:37.260770 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427506 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3170500 -7822> 2013-04-03 10:45:37.260803 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos -7821> 2013-04-03 10:45:37.260828 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394104 -7820> 2013-04-03 10:45:37.260844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos latest full 393798 -7819> 2013-04-03 10:45:37.260863 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos applying incremental log 394105 2013-04-03 10:45:36.175923 mon.0 10.252.24.25:6789/0 178 : [INF] pgmap v29112: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 2847B/s wr, 0op/s -7818> 2013-04-03 10:45:37.260910 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 check_subs -7817> 2013-04-03 10:45:37.260941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 create_pending v 394106 -7816> 2013-04-03 10:45:37.260964 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 _updated_log for mon.0 10.252.24.25:6789/0 -7815> 2013-04-03 10:45:37.260976 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 178) v1 -- ?+0 0x2e8bc40 con 0x1bac160 -7814> 2013-04-03 10:45:37.260996 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos -7813> 2013-04-03 10:45:37.261012 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394105 -7812> 2013-04-03 10:45:37.261020 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7811> 2013-04-03 10:45:37.261031 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 preprocess_log log(1 entries) v1 from mon.0 -7810> 2013-04-03 10:45:37.261049 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 nothing new -7809> 2013-04-03 10:45:37.261100 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos -7808> 2013-04-03 10:45:37.261118 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394105 -7807> 2013-04-03 10:45:37.261136 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7806> 2013-04-03 10:45:37.261158 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 178) v1 ==== 0+0+0 (0 0 0) 0x2e8bc40 con 0x1bac160 -7805> 2013-04-03 10:45:37.261165 7fcdffcdb700 10 handle_log_ack log(last 178) v1 -7804> 2013-04-03 10:45:37.261167 7fcdffcdb700 10 logged 2013-04-03 10:45:36.175923 mon.0 10.252.24.25:6789/0 178 : [INF] pgmap v29112: 240 pgs: 240 active+clean; 256 MB data, 5660 MB used, 1067 GB / 1079 GB avail; 2847B/s wr, 0op/s -7803> 2013-04-03 10:45:37.273995 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113989 ==== paxos(lease_ack lc 427506 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (283821462 0 0) 0x3170500 con 0x208a840 -7802> 2013-04-03 10:45:37.274032 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos -7801> 2013-04-03 10:45:37.274049 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394105 -7800> 2013-04-03 10:45:37.274065 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7799> 2013-04-03 10:45:37.395100 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 183 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21838 v187) v2 ==== 289+0+0 (2255956442 0 0) 0x2f7a080 con 0x200d420 -7798> 2013-04-03 10:45:37.395156 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21838 v187) v2 from mds.? 10.252.24.88:6800/17213 -7797> 2013-04-03 10:45:37.395168 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21838 v187) v2 -- ?+0 0x2145dc0 con 0x200d420 -7796> 2013-04-03 10:45:38.277109 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113990 ==== forward(pg_stats(1 pgs tid 38 v 605) v1) to leader v1 ==== 757+0+0 (2676512919 0 0) 0x2145840 con 0x208a840 -7795> 2013-04-03 10:45:38.277139 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -7794> 2013-04-03 10:45:38.277152 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -7793> 2013-04-03 10:45:38.277216 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 preprocess_query pg_stats(1 pgs tid 38 v 605) v1 from osd.2 10.252.24.30:6801/5656 -7792> 2013-04-03 10:45:38.277241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 prepare_update pg_stats(1 pgs tid 38 v 605) v1 from osd.2 10.252.24.30:6801/5656 -7791> 2013-04-03 10:45:38.277247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 prepare_pg_stats pg_stats(1 pgs tid 38 v 605) v1 from osd.2 -7790> 2013-04-03 10:45:38.277255 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -7789> 2013-04-03 10:45:38.327438 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 encode_pending v 29114 -7788> 2013-04-03 10:45:38.327487 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427506) queue_proposal bl 654 bytes; ctx = 0x2c3dc30 -7787> 2013-04-03 10:45:38.327497 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427506) propose_queued 427507 654 bytes -7786> 2013-04-03 10:45:38.328769 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427506 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3170280 -7785> 2013-04-03 10:45:38.360572 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113991 ==== paxos(accept lc 427506 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (420814146 0 0) 0x3170280 con 0x208a840 -7784> 2013-04-03 10:45:38.360963 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427507 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3170500 -7783> 2013-04-03 10:45:38.360983 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427507 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3170780 -7782> 2013-04-03 10:45:38.361067 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29113 update_from_paxos applying incremental 29114 -7781> 2013-04-03 10:45:38.361089 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 v29114: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 3513B/s wr, 0op/s -7780> 2013-04-03 10:45:38.361108 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 send_pg_creates to 0 pgs -7779> 2013-04-03 10:45:38.361113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 update_logger -7778> 2013-04-03 10:45:38.361124 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 create_pending v 29115 -7777> 2013-04-03 10:45:38.361133 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 _updated_stats for osd.2 10.252.24.30:6801/5656 -7776> 2013-04-03 10:45:38.361141 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 38) v1 tid 17732) v2 -- ?+0 0x24f8000 con 0x208a840 -7775> 2013-04-03 10:45:38.361156 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 check_osd_map already seen 605 >= 605 -7774> 2013-04-03 10:45:38.361159 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 update_logger -7773> 2013-04-03 10:45:38.361170 7fcdffcdb700 0 log [INF] : pgmap v29114: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 3513B/s wr, 0op/s -7772> 2013-04-03 10:45:38.361172 7fcdffcdb700 10 send_log to self -7771> 2013-04-03 10:45:38.361173 7fcdffcdb700 10 log_queue is 2 last_log 180 sent 178 num 2 unsent 2 sending 2 -7770> 2013-04-03 10:45:38.361175 7fcdffcdb700 10 will send 2013-04-03 10:45:37.235969 mon.0 10.252.24.25:6789/0 179 : [INF] pgmap v29113: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail -7769> 2013-04-03 10:45:38.361181 7fcdffcdb700 10 will send 2013-04-03 10:45:38.361171 mon.0 10.252.24.25:6789/0 180 : [INF] pgmap v29114: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 3513B/s wr, 0op/s -7768> 2013-04-03 10:45:38.361185 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x287ec00 -7767> 2013-04-03 10:45:38.361236 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos -7766> 2013-04-03 10:45:38.361259 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394105 -7765> 2013-04-03 10:45:38.361279 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7764> 2013-04-03 10:45:38.361301 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x287ec00 con 0x1bac160 -7763> 2013-04-03 10:45:38.361323 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos -7762> 2013-04-03 10:45:38.361338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394105 -7761> 2013-04-03 10:45:38.361347 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -7760> 2013-04-03 10:45:38.361360 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 preprocess_log log(2 entries) v1 from mon.0 -7759> 2013-04-03 10:45:38.361394 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -7758> 2013-04-03 10:45:38.361408 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 prepare_log log(2 entries) v1 from mon.0 -7757> 2013-04-03 10:45:38.361419 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 logging 2013-04-03 10:45:37.235969 mon.0 10.252.24.25:6789/0 179 : [INF] pgmap v29113: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail -7756> 2013-04-03 10:45:38.361441 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 logging 2013-04-03 10:45:38.361171 mon.0 10.252.24.25:6789/0 180 : [INF] pgmap v29114: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 3513B/s wr, 0op/s -7755> 2013-04-03 10:45:38.375435 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 check_down_pgs -7754> 2013-04-03 10:45:38.375482 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 v29114: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 3513B/s wr, 0op/s -7753> 2013-04-03 10:45:38.375517 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -7752> 2013-04-03 10:45:38.375548 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -7751> 2013-04-03 10:45:38.375589 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -7750> 2013-04-03 10:45:38.375621 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos -7749> 2013-04-03 10:45:38.375641 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394105 -7748> 2013-04-03 10:45:38.375651 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 log -7747> 2013-04-03 10:45:38.375691 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7746> 2013-04-03 10:45:38.375712 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -7745> 2013-04-03 10:45:38.385452 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113992 ==== paxos(lease_ack lc 427507 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3841938276 0 0) 0x3170780 con 0x208a840 -7744> 2013-04-03 10:45:38.385537 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos -7743> 2013-04-03 10:45:38.385558 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 update_from_paxos version 394105 summary v 394105 -7742> 2013-04-03 10:45:38.385576 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7741> 2013-04-03 10:45:39.360786 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394105 encode_pending v394106 -7740> 2013-04-03 10:45:39.360818 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427507) queue_proposal bl 707 bytes; ctx = 0x2c3dd80 -7739> 2013-04-03 10:45:39.360826 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427507) propose_queued 427508 707 bytes -7738> 2013-04-03 10:45:39.361745 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427507 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3170000 -7737> 2013-04-03 10:45:39.386565 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113993 ==== paxos(accept lc 427507 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3590024878 0 0) 0x3170000 con 0x208a840 -7736> 2013-04-03 10:45:39.386922 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427508 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3170780 -7735> 2013-04-03 10:45:39.386944 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427508 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3170280 -7734> 2013-04-03 10:45:39.386984 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos -7733> 2013-04-03 10:45:39.387029 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos version 394106 summary v 394105 -7732> 2013-04-03 10:45:39.387044 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos latest full 393798 -7731> 2013-04-03 10:45:39.387063 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos applying incremental log 394106 2013-04-03 10:45:37.235969 mon.0 10.252.24.25:6789/0 179 : [INF] pgmap v29113: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail -7730> 2013-04-03 10:45:39.387090 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos applying incremental log 394106 2013-04-03 10:45:38.361171 mon.0 10.252.24.25:6789/0 180 : [INF] pgmap v29114: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 3513B/s wr, 0op/s -7729> 2013-04-03 10:45:39.387136 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 check_subs -7728> 2013-04-03 10:45:39.387168 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 create_pending v 394107 -7727> 2013-04-03 10:45:39.387191 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 _updated_log for mon.0 10.252.24.25:6789/0 -7726> 2013-04-03 10:45:39.387205 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 180) v1 -- ?+0 0x2e8ba80 con 0x1bac160 -7725> 2013-04-03 10:45:39.387258 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos -7724> 2013-04-03 10:45:39.387275 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos version 394106 summary v 394106 -7723> 2013-04-03 10:45:39.387293 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7722> 2013-04-03 10:45:39.387317 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 180) v1 ==== 0+0+0 (0 0 0) 0x2e8ba80 con 0x1bac160 -7721> 2013-04-03 10:45:39.387325 7fcdffcdb700 10 handle_log_ack log(last 180) v1 -7720> 2013-04-03 10:45:39.387326 7fcdffcdb700 10 logged 2013-04-03 10:45:37.235969 mon.0 10.252.24.25:6789/0 179 : [INF] pgmap v29113: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail -7719> 2013-04-03 10:45:39.387333 7fcdffcdb700 10 logged 2013-04-03 10:45:38.361171 mon.0 10.252.24.25:6789/0 180 : [INF] pgmap v29114: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 3513B/s wr, 0op/s -7718> 2013-04-03 10:45:39.400220 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113994 ==== paxos(lease_ack lc 427508 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (4197779284 0 0) 0x3170280 con 0x208a840 -7717> 2013-04-03 10:45:39.400304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos -7716> 2013-04-03 10:45:39.400325 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos version 394106 summary v 394106 -7715> 2013-04-03 10:45:39.400343 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7714> 2013-04-03 10:45:39.597247 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 184 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140468 v187) v2 ==== 289+0+0 (1913005241 0 0) 0x2cc5080 con 0x2118420 -7713> 2013-04-03 10:45:39.597334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140468 v187) v2 from mds.? 10.252.58.31:6800/10908 -7712> 2013-04-03 10:45:39.597349 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140468 v187) v2 -- ?+0 0x2145840 con 0x2118420 -7711> 2013-04-03 10:45:40.943931 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113995 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 183 v187) v2) to leader v1 ==== 560+0+0 (3141710737 0 0) 0x2145000 con 0x208a840 -7710> 2013-04-03 10:45:40.943952 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -7709> 2013-04-03 10:45:40.943963 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a51dc0 from 10.252.24.88:6789/0 -7708> 2013-04-03 10:45:40.944018 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 183 v187) v2 from mds.0 10.252.24.25:6802/3144 -7707> 2013-04-03 10:45:40.944031 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 183 v187) v2 tid 17733) v2 -- ?+0 0x24f8b40 con 0x208a840 -7706> 2013-04-03 10:45:41.177255 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2df2280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33169/0 -7705> 2013-04-03 10:45:41.177293 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33169/0) -7704> 2013-04-03 10:45:41.177313 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2280 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -7703> 2013-04-03 10:45:41.177512 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2280 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -7702> 2013-04-03 10:45:41.395256 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 184 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21839 v187) v2 ==== 289+0+0 (2769917607 0 0) 0x2f7ab80 con 0x200d420 -7701> 2013-04-03 10:45:41.395341 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21839 v187) v2 from mds.? 10.252.24.88:6800/17213 -7700> 2013-04-03 10:45:41.395355 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21839 v187) v2 -- ?+0 0x2145000 con 0x200d420 -7699> 2013-04-03 10:45:41.631258 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113996 ==== forward(pg_stats(1 pgs tid 56 v 605) v1) to leader v1 ==== 753+0+0 (3581889896 0 0) 0x2cc5080 con 0x208a840 -7698> 2013-04-03 10:45:41.631283 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -7697> 2013-04-03 10:45:41.631294 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -7696> 2013-04-03 10:45:41.631349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 preprocess_query pg_stats(1 pgs tid 56 v 605) v1 from osd.1 10.252.24.25:6805/3572 -7695> 2013-04-03 10:45:41.631374 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 prepare_update pg_stats(1 pgs tid 56 v 605) v1 from osd.1 10.252.24.25:6805/3572 -7694> 2013-04-03 10:45:41.631380 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 prepare_pg_stats pg_stats(1 pgs tid 56 v 605) v1 from osd.1 -7693> 2013-04-03 10:45:41.631387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -7692> 2013-04-03 10:45:41.681562 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 encode_pending v 29115 -7691> 2013-04-03 10:45:41.681609 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427508) queue_proposal bl 650 bytes; ctx = 0x2c3dd50 -7690> 2013-04-03 10:45:41.681620 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427508) propose_queued 427509 650 bytes -7689> 2013-04-03 10:45:41.682920 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427508 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3170a00 -7688> 2013-04-03 10:45:41.732338 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113997 ==== paxos(accept lc 427508 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (159198006 0 0) 0x3170a00 con 0x208a840 -7687> 2013-04-03 10:45:41.732686 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427509 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3170280 -7686> 2013-04-03 10:45:41.732708 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427509 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3170000 -7685> 2013-04-03 10:45:41.732797 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29114 update_from_paxos applying incremental 29115 -7684> 2013-04-03 10:45:41.732820 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 v29115: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2726B/s wr, 0op/s -7683> 2013-04-03 10:45:41.732842 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 send_pg_creates to 0 pgs -7682> 2013-04-03 10:45:41.732846 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 update_logger -7681> 2013-04-03 10:45:41.732857 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 create_pending v 29116 -7680> 2013-04-03 10:45:41.732867 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 _updated_stats for osd.1 10.252.24.25:6805/3572 -7679> 2013-04-03 10:45:41.732874 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 56) v1 tid 17734) v2 -- ?+0 0x24f8000 con 0x208a840 -7678> 2013-04-03 10:45:41.732885 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 check_osd_map already seen 605 >= 605 -7677> 2013-04-03 10:45:41.732888 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 update_logger -7676> 2013-04-03 10:45:41.732900 7fcdffcdb700 0 log [INF] : pgmap v29115: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2726B/s wr, 0op/s -7675> 2013-04-03 10:45:41.732902 7fcdffcdb700 10 send_log to self -7674> 2013-04-03 10:45:41.732903 7fcdffcdb700 10 log_queue is 1 last_log 181 sent 180 num 1 unsent 1 sending 1 -7673> 2013-04-03 10:45:41.732905 7fcdffcdb700 10 will send 2013-04-03 10:45:41.732901 mon.0 10.252.24.25:6789/0 181 : [INF] pgmap v29115: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2726B/s wr, 0op/s -7672> 2013-04-03 10:45:41.732919 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287ec00 -7671> 2013-04-03 10:45:41.732973 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos -7670> 2013-04-03 10:45:41.732993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos version 394106 summary v 394106 -7669> 2013-04-03 10:45:41.733013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7668> 2013-04-03 10:45:41.733035 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287ec00 con 0x1bac160 -7667> 2013-04-03 10:45:41.733057 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos -7666> 2013-04-03 10:45:41.733072 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos version 394106 summary v 394106 -7665> 2013-04-03 10:45:41.733082 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7664> 2013-04-03 10:45:41.733094 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 preprocess_log log(1 entries) v1 from mon.0 -7663> 2013-04-03 10:45:41.733120 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7662> 2013-04-03 10:45:41.733133 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 prepare_log log(1 entries) v1 from mon.0 -7661> 2013-04-03 10:45:41.733146 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 logging 2013-04-03 10:45:41.732901 mon.0 10.252.24.25:6789/0 181 : [INF] pgmap v29115: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2726B/s wr, 0op/s -7660> 2013-04-03 10:45:41.758500 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113998 ==== paxos(lease_ack lc 427509 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3746862412 0 0) 0x3170000 con 0x208a840 -7659> 2013-04-03 10:45:41.758595 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos -7658> 2013-04-03 10:45:41.758617 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 update_from_paxos version 394106 summary v 394106 -7657> 2013-04-03 10:45:41.758637 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7656> 2013-04-03 10:45:41.797493 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698113999 ==== forward(pg_stats(1 pgs tid 30460 v 605) v1) to leader v1 ==== 757+0+0 (3317389747 0 0) 0x2a51000 con 0x208a840 -7655> 2013-04-03 10:45:41.797521 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -7654> 2013-04-03 10:45:41.797533 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 -7653> 2013-04-03 10:45:41.797589 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 preprocess_query pg_stats(1 pgs tid 30460 v 605) v1 from osd.3 10.252.24.88:6801/17538 -7652> 2013-04-03 10:45:41.797614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 prepare_update pg_stats(1 pgs tid 30460 v 605) v1 from osd.3 10.252.24.88:6801/17538 -7651> 2013-04-03 10:45:41.797619 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 prepare_pg_stats pg_stats(1 pgs tid 30460 v 605) v1 from osd.3 -7650> 2013-04-03 10:45:41.797625 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -7649> 2013-04-03 10:45:42.732569 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394106 encode_pending v394107 -7648> 2013-04-03 10:45:42.732600 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427509) queue_proposal bl 443 bytes; ctx = 0x2c3d9d0 -7647> 2013-04-03 10:45:42.732608 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427509) propose_queued 427510 443 bytes -7646> 2013-04-03 10:45:42.733824 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427509 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3170c80 -7645> 2013-04-03 10:45:42.733925 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 encode_pending v 29116 -7644> 2013-04-03 10:45:42.733959 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427469..427509) queue_proposal bl 654 bytes; ctx = 0x2c3dd40 -7643> 2013-04-03 10:45:42.733967 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427469..427509) propose_new_value not active; proposal queued -7642> 2013-04-03 10:45:42.759586 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114000 ==== paxos(accept lc 427509 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2367871733 0 0) 0x3170c80 con 0x208a840 -7641> 2013-04-03 10:45:42.759942 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427510 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3170000 -7640> 2013-04-03 10:45:42.759965 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427510 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3170a00 -7639> 2013-04-03 10:45:42.760004 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos -7638> 2013-04-03 10:45:42.760044 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos version 394107 summary v 394106 -7637> 2013-04-03 10:45:42.760059 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos latest full 393798 -7636> 2013-04-03 10:45:42.760078 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos applying incremental log 394107 2013-04-03 10:45:41.732901 mon.0 10.252.24.25:6789/0 181 : [INF] pgmap v29115: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2726B/s wr, 0op/s -7635> 2013-04-03 10:45:42.760131 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 check_subs -7634> 2013-04-03 10:45:42.760163 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 create_pending v 394108 -7633> 2013-04-03 10:45:42.760187 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 _updated_log for mon.0 10.252.24.25:6789/0 -7632> 2013-04-03 10:45:42.760200 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 181) v1 -- ?+0 0x2e8b380 con 0x1bac160 -7631> 2013-04-03 10:45:42.760226 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427510) propose_queued 427511 654 bytes -7630> 2013-04-03 10:45:42.760431 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427510 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116a00 -7629> 2013-04-03 10:45:42.760449 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 181) v1 ==== 0+0+0 (0 0 0) 0x2e8b380 con 0x1bac160 -7628> 2013-04-03 10:45:42.760456 7fcdffcdb700 10 handle_log_ack log(last 181) v1 -7627> 2013-04-03 10:45:42.760464 7fcdffcdb700 10 logged 2013-04-03 10:45:41.732901 mon.0 10.252.24.25:6789/0 181 : [INF] pgmap v29115: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2726B/s wr, 0op/s -7626> 2013-04-03 10:45:42.771851 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114001 ==== paxos(lease_ack lc 427510 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (2487854720 0 0) 0x2116a00 con 0x208a840 -7625> 2013-04-03 10:45:42.785718 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114002 ==== paxos(accept lc 427510 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2936122308 0 0) 0x3170a00 con 0x208a840 -7624> 2013-04-03 10:45:42.788703 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427511 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116a00 -7623> 2013-04-03 10:45:42.788725 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427511 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3170c80 -7622> 2013-04-03 10:45:42.788828 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29115 update_from_paxos applying incremental 29116 -7621> 2013-04-03 10:45:42.788850 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 v29116: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s -7620> 2013-04-03 10:45:42.788869 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 send_pg_creates to 0 pgs -7619> 2013-04-03 10:45:42.788874 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 update_logger -7618> 2013-04-03 10:45:42.788885 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 create_pending v 29117 -7617> 2013-04-03 10:45:42.788895 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 _updated_stats for osd.3 10.252.24.88:6801/17538 -7616> 2013-04-03 10:45:42.788902 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30460) v1 tid 17735) v2 -- ?+0 0x24f8b40 con 0x208a840 -7615> 2013-04-03 10:45:42.788913 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 check_osd_map already seen 605 >= 605 -7614> 2013-04-03 10:45:42.788916 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 update_logger -7613> 2013-04-03 10:45:42.788927 7fcdffcdb700 0 log [INF] : pgmap v29116: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s -7612> 2013-04-03 10:45:42.788929 7fcdffcdb700 10 send_log to self -7611> 2013-04-03 10:45:42.788930 7fcdffcdb700 10 log_queue is 1 last_log 182 sent 181 num 1 unsent 1 sending 1 -7610> 2013-04-03 10:45:42.788932 7fcdffcdb700 10 will send 2013-04-03 10:45:42.788928 mon.0 10.252.24.25:6789/0 182 : [INF] pgmap v29116: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s -7609> 2013-04-03 10:45:42.788939 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b76600 -7608> 2013-04-03 10:45:42.788993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos -7607> 2013-04-03 10:45:42.789013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos version 394107 summary v 394107 -7606> 2013-04-03 10:45:42.789032 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7605> 2013-04-03 10:45:42.789054 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76600 con 0x1bac160 -7604> 2013-04-03 10:45:42.789076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos -7603> 2013-04-03 10:45:42.789091 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos version 394107 summary v 394107 -7602> 2013-04-03 10:45:42.789101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7601> 2013-04-03 10:45:42.789112 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 preprocess_log log(1 entries) v1 from mon.0 -7600> 2013-04-03 10:45:42.789135 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7599> 2013-04-03 10:45:42.789147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 prepare_log log(1 entries) v1 from mon.0 -7598> 2013-04-03 10:45:42.789158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 logging 2013-04-03 10:45:42.788928 mon.0 10.252.24.25:6789/0 182 : [INF] pgmap v29116: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s -7597> 2013-04-03 10:45:42.837836 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114003 ==== paxos(lease_ack lc 427511 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3843697114 0 0) 0x3170c80 con 0x208a840 -7596> 2013-04-03 10:45:42.837928 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos -7595> 2013-04-03 10:45:42.837949 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos version 394107 summary v 394107 -7594> 2013-04-03 10:45:42.837967 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7593> 2013-04-03 10:45:43.277826 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114004 ==== forward(pg_stats(1 pgs tid 39 v 605) v1) to leader v1 ==== 757+0+0 (3031808465 0 0) 0x2a51b00 con 0x208a840 -7592> 2013-04-03 10:45:43.277854 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -7591> 2013-04-03 10:45:43.277866 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -7590> 2013-04-03 10:45:43.277919 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 preprocess_query pg_stats(1 pgs tid 39 v 605) v1 from osd.2 10.252.24.30:6801/5656 -7589> 2013-04-03 10:45:43.277945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 prepare_update pg_stats(1 pgs tid 39 v 605) v1 from osd.2 10.252.24.30:6801/5656 -7588> 2013-04-03 10:45:43.277952 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 prepare_pg_stats pg_stats(1 pgs tid 39 v 605) v1 from osd.2 -7587> 2013-04-03 10:45:43.277959 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -7586> 2013-04-03 10:45:43.375846 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 check_down_pgs -7585> 2013-04-03 10:45:43.375881 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 v29116: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s -7584> 2013-04-03 10:45:43.375913 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -7583> 2013-04-03 10:45:43.375941 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -7582> 2013-04-03 10:45:43.375979 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -7581> 2013-04-03 10:45:43.376010 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos -7580> 2013-04-03 10:45:43.376030 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 update_from_paxos version 394107 summary v 394107 -7579> 2013-04-03 10:45:43.376040 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 log -7578> 2013-04-03 10:45:43.376057 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7577> 2013-04-03 10:45:43.376076 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -7576> 2013-04-03 10:45:43.597371 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 185 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140469 v187) v2 ==== 289+0+0 (1365810116 0 0) 0x2cc4580 con 0x2118420 -7575> 2013-04-03 10:45:43.597452 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140469 v187) v2 from mds.? 10.252.58.31:6800/10908 -7574> 2013-04-03 10:45:43.597467 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140469 v187) v2 -- ?+0 0x2a51b00 con 0x2118420 -7573> 2013-04-03 10:45:43.785913 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394107 encode_pending v394108 -7572> 2013-04-03 10:45:43.785955 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427511) queue_proposal bl 443 bytes; ctx = 0x2c3dd60 -7571> 2013-04-03 10:45:43.785963 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427511) propose_queued 427512 443 bytes -7570> 2013-04-03 10:45:43.787277 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427511 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003000 -7569> 2013-04-03 10:45:43.787360 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 encode_pending v 29117 -7568> 2013-04-03 10:45:43.787395 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427469..427511) queue_proposal bl 654 bytes; ctx = 0x2c3dd30 -7567> 2013-04-03 10:45:43.787403 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427469..427511) propose_new_value not active; proposal queued -7566> 2013-04-03 10:45:43.817814 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114005 ==== paxos(accept lc 427511 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1641056309 0 0) 0x3003000 con 0x208a840 -7565> 2013-04-03 10:45:43.818198 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427512 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3170c80 -7564> 2013-04-03 10:45:43.818221 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427512 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3170a00 -7563> 2013-04-03 10:45:43.818266 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos -7562> 2013-04-03 10:45:43.818305 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos version 394108 summary v 394107 -7561> 2013-04-03 10:45:43.818320 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos latest full 393798 -7560> 2013-04-03 10:45:43.818342 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos applying incremental log 394108 2013-04-03 10:45:42.788928 mon.0 10.252.24.25:6789/0 182 : [INF] pgmap v29116: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s -7559> 2013-04-03 10:45:43.818393 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 check_subs -7558> 2013-04-03 10:45:43.818426 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 create_pending v 394109 -7557> 2013-04-03 10:45:43.818449 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 _updated_log for mon.0 10.252.24.25:6789/0 -7556> 2013-04-03 10:45:43.818463 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 182) v1 -- ?+0 0x2e8b700 con 0x1bac160 -7555> 2013-04-03 10:45:43.818489 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427512) propose_queued 427513 654 bytes -7554> 2013-04-03 10:45:43.818690 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427512 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4a00 -7553> 2013-04-03 10:45:43.818711 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 182) v1 ==== 0+0+0 (0 0 0) 0x2e8b700 con 0x1bac160 -7552> 2013-04-03 10:45:43.818718 7fcdffcdb700 10 handle_log_ack log(last 182) v1 -7551> 2013-04-03 10:45:43.818720 7fcdffcdb700 10 logged 2013-04-03 10:45:42.788928 mon.0 10.252.24.25:6789/0 182 : [INF] pgmap v29116: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1161B/s wr, 0op/s -7550> 2013-04-03 10:45:43.831014 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114006 ==== paxos(lease_ack lc 427512 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (2399705424 0 0) 0x29b4a00 con 0x208a840 -7549> 2013-04-03 10:45:43.842954 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114007 ==== paxos(accept lc 427512 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (870804041 0 0) 0x3170a00 con 0x208a840 -7548> 2013-04-03 10:45:43.843388 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427513 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4a00 -7547> 2013-04-03 10:45:43.843410 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427513 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3003000 -7546> 2013-04-03 10:45:43.843499 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29116 update_from_paxos applying incremental 29117 -7545> 2013-04-03 10:45:43.843521 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 v29117: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2431B/s wr, 0op/s -7544> 2013-04-03 10:45:43.843540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 send_pg_creates to 0 pgs -7543> 2013-04-03 10:45:43.843545 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 update_logger -7542> 2013-04-03 10:45:43.843556 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 create_pending v 29118 -7541> 2013-04-03 10:45:43.843566 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 _updated_stats for osd.2 10.252.24.30:6801/5656 -7540> 2013-04-03 10:45:43.843573 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 39) v1 tid 17736) v2 -- ?+0 0x24f8000 con 0x208a840 -7539> 2013-04-03 10:45:43.843583 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 check_osd_map already seen 605 >= 605 -7538> 2013-04-03 10:45:43.843587 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 update_logger -7537> 2013-04-03 10:45:43.843598 7fcdffcdb700 0 log [INF] : pgmap v29117: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2431B/s wr, 0op/s -7536> 2013-04-03 10:45:43.843599 7fcdffcdb700 10 send_log to self -7535> 2013-04-03 10:45:43.843600 7fcdffcdb700 10 log_queue is 1 last_log 183 sent 182 num 1 unsent 1 sending 1 -7534> 2013-04-03 10:45:43.843611 7fcdffcdb700 10 will send 2013-04-03 10:45:43.843599 mon.0 10.252.24.25:6789/0 183 : [INF] pgmap v29117: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2431B/s wr, 0op/s -7533> 2013-04-03 10:45:43.843619 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24e8c00 -7532> 2013-04-03 10:45:43.843672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos -7531> 2013-04-03 10:45:43.843692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos version 394108 summary v 394108 -7530> 2013-04-03 10:45:43.843711 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7529> 2013-04-03 10:45:43.843735 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24e8c00 con 0x1bac160 -7528> 2013-04-03 10:45:43.843757 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos -7527> 2013-04-03 10:45:43.843772 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos version 394108 summary v 394108 -7526> 2013-04-03 10:45:43.843782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7525> 2013-04-03 10:45:43.843794 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 preprocess_log log(1 entries) v1 from mon.0 -7524> 2013-04-03 10:45:43.843819 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7523> 2013-04-03 10:45:43.843833 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 prepare_log log(1 entries) v1 from mon.0 -7522> 2013-04-03 10:45:43.843844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 logging 2013-04-03 10:45:43.843599 mon.0 10.252.24.25:6789/0 183 : [INF] pgmap v29117: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2431B/s wr, 0op/s -7521> 2013-04-03 10:45:43.868363 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114008 ==== paxos(lease_ack lc 427513 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (378648318 0 0) 0x3003000 con 0x208a840 -7520> 2013-04-03 10:45:43.868451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos -7519> 2013-04-03 10:45:43.868472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 update_from_paxos version 394108 summary v 394108 -7518> 2013-04-03 10:45:43.868490 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7517> 2013-04-03 10:45:44.843156 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394108 encode_pending v394109 -7516> 2013-04-03 10:45:44.843186 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427513) queue_proposal bl 443 bytes; ctx = 0x2c3d790 -7515> 2013-04-03 10:45:44.843195 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427513) propose_queued 427514 443 bytes -7514> 2013-04-03 10:45:44.844118 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427513 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003280 -7513> 2013-04-03 10:45:44.867689 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114009 ==== paxos(accept lc 427513 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3274364015 0 0) 0x3003280 con 0x208a840 -7512> 2013-04-03 10:45:44.868049 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427514 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003000 -7511> 2013-04-03 10:45:44.868072 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427514 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3170a00 -7510> 2013-04-03 10:45:44.868117 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos -7509> 2013-04-03 10:45:44.868156 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos version 394109 summary v 394108 -7508> 2013-04-03 10:45:44.868171 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos latest full 393798 -7507> 2013-04-03 10:45:44.868190 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos applying incremental log 394109 2013-04-03 10:45:43.843599 mon.0 10.252.24.25:6789/0 183 : [INF] pgmap v29117: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2431B/s wr, 0op/s -7506> 2013-04-03 10:45:44.868244 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 check_subs -7505> 2013-04-03 10:45:44.868277 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 create_pending v 394110 -7504> 2013-04-03 10:45:44.868300 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 _updated_log for mon.0 10.252.24.25:6789/0 -7503> 2013-04-03 10:45:44.868314 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 183) v1 -- ?+0 0x2e8b700 con 0x1bac160 -7502> 2013-04-03 10:45:44.868366 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos -7501> 2013-04-03 10:45:44.868383 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos version 394109 summary v 394109 -7500> 2013-04-03 10:45:44.868402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7499> 2013-04-03 10:45:44.868425 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 183) v1 ==== 0+0+0 (0 0 0) 0x2e8b700 con 0x1bac160 -7498> 2013-04-03 10:45:44.868434 7fcdffcdb700 10 handle_log_ack log(last 183) v1 -7497> 2013-04-03 10:45:44.868435 7fcdffcdb700 10 logged 2013-04-03 10:45:43.843599 mon.0 10.252.24.25:6789/0 183 : [INF] pgmap v29117: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2431B/s wr, 0op/s -7496> 2013-04-03 10:45:44.879754 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114010 ==== paxos(lease_ack lc 427514 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3778576505 0 0) 0x3170a00 con 0x208a840 -7495> 2013-04-03 10:45:44.879839 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos -7494> 2013-04-03 10:45:44.879861 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos version 394109 summary v 394109 -7493> 2013-04-03 10:45:44.879878 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7492> 2013-04-03 10:45:44.944085 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114011 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 184 v187) v2) to leader v1 ==== 560+0+0 (466453206 0 0) 0x2a52600 con 0x208a840 -7491> 2013-04-03 10:45:44.944106 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -7490> 2013-04-03 10:45:44.944116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a52b80 from 10.252.24.88:6789/0 -7489> 2013-04-03 10:45:44.944169 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 184 v187) v2 from mds.0 10.252.24.25:6802/3144 -7488> 2013-04-03 10:45:44.944181 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 184 v187) v2 tid 17737) v2 -- ?+0 0x24f8b40 con 0x208a840 -7487> 2013-04-03 10:45:45.395362 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 185 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21840 v187) v2 ==== 289+0+0 (1857699327 0 0) 0x2f7a600 con 0x200d420 -7486> 2013-04-03 10:45:45.395442 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21840 v187) v2 from mds.? 10.252.24.88:6800/17213 -7485> 2013-04-03 10:45:45.395457 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21840 v187) v2 -- ?+0 0x2a52600 con 0x200d420 -7484> 2013-04-03 10:45:46.631816 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114012 ==== forward(pg_stats(1 pgs tid 57 v 605) v1) to leader v1 ==== 753+0+0 (1255930587 0 0) 0x2cc4580 con 0x208a840 -7483> 2013-04-03 10:45:46.631838 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -7482> 2013-04-03 10:45:46.631849 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -7481> 2013-04-03 10:45:46.631906 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 preprocess_query pg_stats(1 pgs tid 57 v 605) v1 from osd.1 10.252.24.25:6805/3572 -7480> 2013-04-03 10:45:46.631932 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 prepare_update pg_stats(1 pgs tid 57 v 605) v1 from osd.1 10.252.24.25:6805/3572 -7479> 2013-04-03 10:45:46.631938 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 prepare_pg_stats pg_stats(1 pgs tid 57 v 605) v1 from osd.1 -7478> 2013-04-03 10:45:46.631945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -7477> 2013-04-03 10:45:46.682139 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 encode_pending v 29118 -7476> 2013-04-03 10:45:46.682180 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427514) queue_proposal bl 650 bytes; ctx = 0x2c3dd20 -7475> 2013-04-03 10:45:46.682188 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427514) propose_queued 427515 650 bytes -7474> 2013-04-03 10:45:46.683354 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427514 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003500 -7473> 2013-04-03 10:45:46.711652 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114013 ==== paxos(accept lc 427514 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (834969413 0 0) 0x3003500 con 0x208a840 -7472> 2013-04-03 10:45:46.712566 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427515 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3170a00 -7471> 2013-04-03 10:45:46.712587 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427515 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3003280 -7470> 2013-04-03 10:45:46.712676 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29117 update_from_paxos applying incremental 29118 -7469> 2013-04-03 10:45:46.712701 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 v29118: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2074B/s wr, 0op/s -7468> 2013-04-03 10:45:46.712721 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 send_pg_creates to 0 pgs -7467> 2013-04-03 10:45:46.712725 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 update_logger -7466> 2013-04-03 10:45:46.712736 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 create_pending v 29119 -7465> 2013-04-03 10:45:46.712746 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 _updated_stats for osd.1 10.252.24.25:6805/3572 -7464> 2013-04-03 10:45:46.712754 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 57) v1 tid 17738) v2 -- ?+0 0x24f8000 con 0x208a840 -7463> 2013-04-03 10:45:46.712769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 check_osd_map already seen 605 >= 605 -7462> 2013-04-03 10:45:46.712773 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 update_logger -7461> 2013-04-03 10:45:46.712783 7fcdffcdb700 0 log [INF] : pgmap v29118: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2074B/s wr, 0op/s -7460> 2013-04-03 10:45:46.712785 7fcdffcdb700 10 send_log to self -7459> 2013-04-03 10:45:46.712786 7fcdffcdb700 10 log_queue is 1 last_log 184 sent 183 num 1 unsent 1 sending 1 -7458> 2013-04-03 10:45:46.712788 7fcdffcdb700 10 will send 2013-04-03 10:45:46.712785 mon.0 10.252.24.25:6789/0 184 : [INF] pgmap v29118: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2074B/s wr, 0op/s -7457> 2013-04-03 10:45:46.712795 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24e8c00 -7456> 2013-04-03 10:45:46.712847 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos -7455> 2013-04-03 10:45:46.712867 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos version 394109 summary v 394109 -7454> 2013-04-03 10:45:46.712886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7453> 2013-04-03 10:45:46.712908 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24e8c00 con 0x1bac160 -7452> 2013-04-03 10:45:46.712930 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos -7451> 2013-04-03 10:45:46.712945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos version 394109 summary v 394109 -7450> 2013-04-03 10:45:46.712956 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7449> 2013-04-03 10:45:46.712968 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 preprocess_log log(1 entries) v1 from mon.0 -7448> 2013-04-03 10:45:46.712992 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7447> 2013-04-03 10:45:46.713005 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 prepare_log log(1 entries) v1 from mon.0 -7446> 2013-04-03 10:45:46.713015 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 logging 2013-04-03 10:45:46.712785 mon.0 10.252.24.25:6789/0 184 : [INF] pgmap v29118: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2074B/s wr, 0op/s -7445> 2013-04-03 10:45:46.724921 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114014 ==== paxos(lease_ack lc 427515 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3015787812 0 0) 0x3003280 con 0x208a840 -7444> 2013-04-03 10:45:46.724974 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos -7443> 2013-04-03 10:45:46.724993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 update_from_paxos version 394109 summary v 394109 -7442> 2013-04-03 10:45:46.725009 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7441> 2013-04-03 10:45:47.597520 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 186 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140470 v187) v2 ==== 289+0+0 (886790723 0 0) 0x2cc4000 con 0x2118420 -7440> 2013-04-03 10:45:47.597603 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140470 v187) v2 from mds.? 10.252.58.31:6800/10908 -7439> 2013-04-03 10:45:47.597618 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140470 v187) v2 -- ?+0 0x2cc4580 con 0x2118420 -7438> 2013-04-03 10:45:47.711845 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394109 encode_pending v394110 -7437> 2013-04-03 10:45:47.711876 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427515) queue_proposal bl 443 bytes; ctx = 0x2c3d9e0 -7436> 2013-04-03 10:45:47.711884 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427515) propose_queued 427516 443 bytes -7435> 2013-04-03 10:45:47.712832 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427515 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116780 -7434> 2013-04-03 10:45:47.852926 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114015 ==== paxos(accept lc 427515 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4111827252 0 0) 0x2116780 con 0x208a840 -7433> 2013-04-03 10:45:47.853254 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427516 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003280 -7432> 2013-04-03 10:45:47.853274 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427516 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3003500 -7431> 2013-04-03 10:45:47.853316 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos -7430> 2013-04-03 10:45:47.853355 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394109 -7429> 2013-04-03 10:45:47.853370 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos latest full 393798 -7428> 2013-04-03 10:45:47.853389 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos applying incremental log 394110 2013-04-03 10:45:46.712785 mon.0 10.252.24.25:6789/0 184 : [INF] pgmap v29118: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2074B/s wr, 0op/s -7427> 2013-04-03 10:45:47.853443 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 check_subs -7426> 2013-04-03 10:45:47.853475 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 create_pending v 394111 -7425> 2013-04-03 10:45:47.853498 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 _updated_log for mon.0 10.252.24.25:6789/0 -7424> 2013-04-03 10:45:47.853512 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 184) v1 -- ?+0 0x2e8b1c0 con 0x1bac160 -7423> 2013-04-03 10:45:47.853565 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos -7422> 2013-04-03 10:45:47.853581 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 -7421> 2013-04-03 10:45:47.853600 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7420> 2013-04-03 10:45:47.853622 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 184) v1 ==== 0+0+0 (0 0 0) 0x2e8b1c0 con 0x1bac160 -7419> 2013-04-03 10:45:47.853631 7fcdffcdb700 10 handle_log_ack log(last 184) v1 -7418> 2013-04-03 10:45:47.853632 7fcdffcdb700 10 logged 2013-04-03 10:45:46.712785 mon.0 10.252.24.25:6789/0 184 : [INF] pgmap v29118: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2074B/s wr, 0op/s -7417> 2013-04-03 10:45:47.864984 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114016 ==== paxos(lease_ack lc 427516 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (407790034 0 0) 0x3003500 con 0x208a840 -7416> 2013-04-03 10:45:47.865069 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos -7415> 2013-04-03 10:45:47.865089 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 -7414> 2013-04-03 10:45:47.865107 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7413> 2013-04-03 10:45:48.376199 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 check_down_pgs -7412> 2013-04-03 10:45:48.376232 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 v29118: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2074B/s wr, 0op/s -7411> 2013-04-03 10:45:48.376262 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -7410> 2013-04-03 10:45:48.376288 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -7409> 2013-04-03 10:45:48.376326 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -7408> 2013-04-03 10:45:48.376360 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos -7407> 2013-04-03 10:45:48.376379 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 -7406> 2013-04-03 10:45:48.376389 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 log -7405> 2013-04-03 10:45:48.376406 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7404> 2013-04-03 10:45:48.376425 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -7403> 2013-04-03 10:45:48.944158 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114017 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 185 v187) v2) to leader v1 ==== 560+0+0 (3333982083 0 0) 0x2a52080 con 0x208a840 -7402> 2013-04-03 10:45:48.944181 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -7401> 2013-04-03 10:45:48.944193 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a512c0 from 10.252.24.88:6789/0 -7400> 2013-04-03 10:45:48.944249 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 185 v187) v2 from mds.0 10.252.24.25:6802/3144 -7399> 2013-04-03 10:45:48.944262 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 185 v187) v2 tid 17739) v2 -- ?+0 0x24f8b40 con 0x208a840 -7398> 2013-04-03 10:45:49.395457 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 186 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21841 v187) v2 ==== 289+0+0 (1305772674 0 0) 0x2f79dc0 con 0x200d420 -7397> 2013-04-03 10:45:49.395540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21841 v187) v2 from mds.? 10.252.24.88:6800/17213 -7396> 2013-04-03 10:45:49.395555 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21841 v187) v2 -- ?+0 0x2a52080 con 0x200d420 -7395> 2013-04-03 10:45:50.853337 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427516 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3062280 -7394> 2013-04-03 10:45:50.854162 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114018 ==== paxos(lease_ack lc 427516 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (166009523 0 0) 0x3062280 con 0x208a840 -7393> 2013-04-03 10:45:50.854234 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos -7392> 2013-04-03 10:45:50.854255 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 -7391> 2013-04-03 10:45:50.854273 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7390> 2013-04-03 10:45:51.193269 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2df2500 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33170/0 -7389> 2013-04-03 10:45:51.193338 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2500 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33170/0) -7388> 2013-04-03 10:45:51.193357 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2500 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -7387> 2013-04-03 10:45:51.193522 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2df2500 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -7386> 2013-04-03 10:45:51.597628 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 187 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140471 v187) v2 ==== 289+0+0 (397796670 0 0) 0x2cc5600 con 0x2118420 -7385> 2013-04-03 10:45:51.597712 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140471 v187) v2 from mds.? 10.252.58.31:6800/10908 -7384> 2013-04-03 10:45:51.597727 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140471 v187) v2 -- ?+0 0x2f79dc0 con 0x2118420 -7383> 2013-04-03 10:45:52.944304 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114019 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 186 v187) v2) to leader v1 ==== 560+0+0 (3906975679 0 0) 0x2cc4000 con 0x208a840 -7382> 2013-04-03 10:45:52.944328 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -7381> 2013-04-03 10:45:52.944339 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a51840 from 10.252.24.88:6789/0 -7380> 2013-04-03 10:45:52.944397 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 186 v187) v2 from mds.0 10.252.24.25:6802/3144 -7379> 2013-04-03 10:45:52.944408 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 186 v187) v2 tid 17740) v2 -- ?+0 0x24f8000 con 0x208a840 -7378> 2013-04-03 10:45:53.376550 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 v29118: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2074B/s wr, 0op/s -7377> 2013-04-03 10:45:53.376593 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -7376> 2013-04-03 10:45:53.376617 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -7375> 2013-04-03 10:45:53.376665 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -7374> 2013-04-03 10:45:53.376699 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos -7373> 2013-04-03 10:45:53.376719 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 -7372> 2013-04-03 10:45:53.376729 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 log -7371> 2013-04-03 10:45:53.376746 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7370> 2013-04-03 10:45:53.376765 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -7369> 2013-04-03 10:45:53.395548 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 187 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21842 v187) v2 ==== 289+0+0 (677856005 0 0) 0x2f792c0 con 0x200d420 -7368> 2013-04-03 10:45:53.395630 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21842 v187) v2 from mds.? 10.252.24.88:6800/17213 -7367> 2013-04-03 10:45:53.395645 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21842 v187) v2 -- ?+0 0x2cc4000 con 0x200d420 -7366> 2013-04-03 10:45:53.853417 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427516 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3062780 -7365> 2013-04-03 10:45:53.854218 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114020 ==== paxos(lease_ack lc 427516 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3427357532 0 0) 0x3062780 con 0x208a840 -7364> 2013-04-03 10:45:53.854299 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos -7363> 2013-04-03 10:45:53.854321 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 -7362> 2013-04-03 10:45:53.854338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7361> 2013-04-03 10:45:55.597721 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 188 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140472 v187) v2 ==== 289+0+0 (3777320020 0 0) 0x273b080 con 0x2118420 -7360> 2013-04-03 10:45:55.597802 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140472 v187) v2 from mds.? 10.252.58.31:6800/10908 -7359> 2013-04-03 10:45:55.597824 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140472 v187) v2 -- ?+0 0x2f792c0 con 0x2118420 -7358> 2013-04-03 10:45:56.632969 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114021 ==== forward(pg_stats(1 pgs tid 58 v 605) v1) to leader v1 ==== 753+0+0 (68721748 0 0) 0x2cc5600 con 0x208a840 -7357> 2013-04-03 10:45:56.632994 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -7356> 2013-04-03 10:45:56.633010 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 -7355> 2013-04-03 10:45:56.633080 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 preprocess_query pg_stats(1 pgs tid 58 v 605) v1 from osd.1 10.252.24.25:6805/3572 -7354> 2013-04-03 10:45:56.633107 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 prepare_update pg_stats(1 pgs tid 58 v 605) v1 from osd.1 10.252.24.25:6805/3572 -7353> 2013-04-03 10:45:56.633112 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 prepare_pg_stats pg_stats(1 pgs tid 58 v 605) v1 from osd.1 -7352> 2013-04-03 10:45:56.633121 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -7351> 2013-04-03 10:45:56.683332 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 encode_pending v 29119 -7350> 2013-04-03 10:45:56.683371 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427516) queue_proposal bl 650 bytes; ctx = 0x2c3dc20 -7349> 2013-04-03 10:45:56.683379 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427516) propose_queued 427517 650 bytes -7348> 2013-04-03 10:45:56.684614 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427516 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062500 -7347> 2013-04-03 10:45:56.725540 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114022 ==== paxos(accept lc 427516 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1429443641 0 0) 0x3062500 con 0x208a840 -7346> 2013-04-03 10:45:56.725888 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427517 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062780 -7345> 2013-04-03 10:45:56.725914 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427517 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3062280 -7344> 2013-04-03 10:45:56.726006 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29118 update_from_paxos applying incremental 29119 -7343> 2013-04-03 10:45:56.726028 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 v29119: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 635B/s wr, 0op/s -7342> 2013-04-03 10:45:56.726049 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 send_pg_creates to 0 pgs -7341> 2013-04-03 10:45:56.726054 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 update_logger -7340> 2013-04-03 10:45:56.726065 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 create_pending v 29120 -7339> 2013-04-03 10:45:56.726074 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 _updated_stats for osd.1 10.252.24.25:6805/3572 -7338> 2013-04-03 10:45:56.726081 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 58) v1 tid 17741) v2 -- ?+0 0x24f8b40 con 0x208a840 -7337> 2013-04-03 10:45:56.726091 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 check_osd_map already seen 605 >= 605 -7336> 2013-04-03 10:45:56.726095 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 update_logger -7335> 2013-04-03 10:45:56.726106 7fcdffcdb700 0 log [INF] : pgmap v29119: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 635B/s wr, 0op/s -7334> 2013-04-03 10:45:56.726108 7fcdffcdb700 10 send_log to self -7333> 2013-04-03 10:45:56.726108 7fcdffcdb700 10 log_queue is 1 last_log 185 sent 184 num 1 unsent 1 sending 1 -7332> 2013-04-03 10:45:56.726110 7fcdffcdb700 10 will send 2013-04-03 10:45:56.726107 mon.0 10.252.24.25:6789/0 185 : [INF] pgmap v29119: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 635B/s wr, 0op/s -7331> 2013-04-03 10:45:56.726117 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24e8c00 -7330> 2013-04-03 10:45:56.726170 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos -7329> 2013-04-03 10:45:56.726190 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 -7328> 2013-04-03 10:45:56.726209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7327> 2013-04-03 10:45:56.726231 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24e8c00 con 0x1bac160 -7326> 2013-04-03 10:45:56.726256 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos -7325> 2013-04-03 10:45:56.726271 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 -7324> 2013-04-03 10:45:56.726281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7323> 2013-04-03 10:45:56.726293 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 preprocess_log log(1 entries) v1 from mon.0 -7322> 2013-04-03 10:45:56.726318 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7321> 2013-04-03 10:45:56.726331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 prepare_log log(1 entries) v1 from mon.0 -7320> 2013-04-03 10:45:56.726343 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 logging 2013-04-03 10:45:56.726107 mon.0 10.252.24.25:6789/0 185 : [INF] pgmap v29119: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 635B/s wr, 0op/s -7319> 2013-04-03 10:45:56.737539 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114023 ==== paxos(lease_ack lc 427517 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (1344398760 0 0) 0x3062280 con 0x208a840 -7318> 2013-04-03 10:45:56.737631 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos -7317> 2013-04-03 10:45:56.737653 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 update_from_paxos version 394110 summary v 394110 -7316> 2013-04-03 10:45:56.737672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7315> 2013-04-03 10:45:56.944540 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114024 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 187 v187) v2) to leader v1 ==== 560+0+0 (900379370 0 0) 0x2a528c0 con 0x208a840 -7314> 2013-04-03 10:45:56.944562 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -7313> 2013-04-03 10:45:56.944572 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc4b00 from 10.252.24.88:6789/0 -7312> 2013-04-03 10:45:56.944627 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 187 v187) v2 from mds.0 10.252.24.25:6802/3144 -7311> 2013-04-03 10:45:56.944639 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 187 v187) v2 tid 17742) v2 -- ?+0 0x24f8000 con 0x208a840 -7310> 2013-04-03 10:45:57.395637 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 188 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21843 v187) v2 ==== 289+0+0 (185203832 0 0) 0x2f7a340 con 0x200d420 -7309> 2013-04-03 10:45:57.395712 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21843 v187) v2 from mds.? 10.252.24.88:6800/17213 -7308> 2013-04-03 10:45:57.395726 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21843 v187) v2 -- ?+0 0x2a528c0 con 0x200d420 -7307> 2013-04-03 10:45:57.725765 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394110 encode_pending v394111 -7306> 2013-04-03 10:45:57.725799 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427517) queue_proposal bl 442 bytes; ctx = 0x2c3dc10 -7305> 2013-04-03 10:45:57.725807 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427517) propose_queued 427518 442 bytes -7304> 2013-04-03 10:45:57.727066 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427517 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4a00 -7303> 2013-04-03 10:45:57.754042 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114025 ==== paxos(accept lc 427517 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (772602038 0 0) 0x20f4a00 con 0x208a840 -7302> 2013-04-03 10:45:57.754370 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427518 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062280 -7301> 2013-04-03 10:45:57.754390 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427518 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3062500 -7300> 2013-04-03 10:45:57.754430 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos -7299> 2013-04-03 10:45:57.754468 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos version 394111 summary v 394110 -7298> 2013-04-03 10:45:57.754483 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos latest full 393798 -7297> 2013-04-03 10:45:57.754501 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos applying incremental log 394111 2013-04-03 10:45:56.726107 mon.0 10.252.24.25:6789/0 185 : [INF] pgmap v29119: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 635B/s wr, 0op/s -7296> 2013-04-03 10:45:57.754553 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 check_subs -7295> 2013-04-03 10:45:57.754586 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 create_pending v 394112 -7294> 2013-04-03 10:45:57.754609 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 _updated_log for mon.0 10.252.24.25:6789/0 -7293> 2013-04-03 10:45:57.754623 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 185) v1 -- ?+0 0x28a5540 con 0x1bac160 -7292> 2013-04-03 10:45:57.754676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos -7291> 2013-04-03 10:45:57.754692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos version 394111 summary v 394111 -7290> 2013-04-03 10:45:57.754711 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7289> 2013-04-03 10:45:57.754733 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 185) v1 ==== 0+0+0 (0 0 0) 0x28a5540 con 0x1bac160 -7288> 2013-04-03 10:45:57.754741 7fcdffcdb700 10 handle_log_ack log(last 185) v1 -7287> 2013-04-03 10:45:57.754743 7fcdffcdb700 10 logged 2013-04-03 10:45:56.726107 mon.0 10.252.24.25:6789/0 185 : [INF] pgmap v29119: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 635B/s wr, 0op/s -7286> 2013-04-03 10:45:57.779407 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114026 ==== paxos(lease_ack lc 427518 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (1631269188 0 0) 0x3062500 con 0x208a840 -7285> 2013-04-03 10:45:57.779455 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos -7284> 2013-04-03 10:45:57.779476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos version 394111 summary v 394111 -7283> 2013-04-03 10:45:57.779492 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7282> 2013-04-03 10:45:58.279519 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114027 ==== forward(pg_stats(1 pgs tid 40 v 605) v1) to leader v1 ==== 757+0+0 (1879574896 0 0) 0x2cc5600 con 0x208a840 -7281> 2013-04-03 10:45:58.279548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -7280> 2013-04-03 10:45:58.279560 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 -7279> 2013-04-03 10:45:58.279613 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 preprocess_query pg_stats(1 pgs tid 40 v 605) v1 from osd.2 10.252.24.30:6801/5656 -7278> 2013-04-03 10:45:58.279638 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 prepare_update pg_stats(1 pgs tid 40 v 605) v1 from osd.2 10.252.24.30:6801/5656 -7277> 2013-04-03 10:45:58.279645 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 prepare_pg_stats pg_stats(1 pgs tid 40 v 605) v1 from osd.2 -7276> 2013-04-03 10:45:58.279652 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -7275> 2013-04-03 10:45:58.376923 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 check_down_pgs -7274> 2013-04-03 10:45:58.376985 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 v29119: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 635B/s wr, 0op/s -7273> 2013-04-03 10:45:58.377022 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -7272> 2013-04-03 10:45:58.377051 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -7271> 2013-04-03 10:45:58.377090 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -7270> 2013-04-03 10:45:58.377123 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos -7269> 2013-04-03 10:45:58.377143 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos version 394111 summary v 394111 -7268> 2013-04-03 10:45:58.377153 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 log -7267> 2013-04-03 10:45:58.377171 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7266> 2013-04-03 10:45:58.377191 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -7265> 2013-04-03 10:45:58.754236 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 encode_pending v 29120 -7264> 2013-04-03 10:45:58.754284 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427518) queue_proposal bl 654 bytes; ctx = 0x2c3d900 -7263> 2013-04-03 10:45:58.754292 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427518) propose_queued 427519 654 bytes -7262> 2013-04-03 10:45:58.755552 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427518 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4280 -7261> 2013-04-03 10:45:58.787026 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114028 ==== paxos(accept lc 427518 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2616852593 0 0) 0x20f4280 con 0x208a840 -7260> 2013-04-03 10:45:58.793653 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427519 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062500 -7259> 2013-04-03 10:45:58.793673 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427519 fc 427469 pn 0 opn 0) v3 -- ?+0 0x20f4a00 -7258> 2013-04-03 10:45:58.793765 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29119 update_from_paxos applying incremental 29120 -7257> 2013-04-03 10:45:58.793788 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29120 v29120: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1696B/s wr, 0op/s -7256> 2013-04-03 10:45:58.793807 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29120 send_pg_creates to 0 pgs -7255> 2013-04-03 10:45:58.793818 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29120 update_logger -7254> 2013-04-03 10:45:58.793831 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29120 create_pending v 29121 -7253> 2013-04-03 10:45:58.793845 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29120 _updated_stats for osd.2 10.252.24.30:6801/5656 -7252> 2013-04-03 10:45:58.793853 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 40) v1 tid 17743) v2 -- ?+0 0x24f8b40 con 0x208a840 -7251> 2013-04-03 10:45:58.793865 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29120 check_osd_map already seen 605 >= 605 -7250> 2013-04-03 10:45:58.793869 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29120 update_logger -7249> 2013-04-03 10:45:58.793881 7fcdffcdb700 0 log [INF] : pgmap v29120: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1696B/s wr, 0op/s -7248> 2013-04-03 10:45:58.793882 7fcdffcdb700 10 send_log to self -7247> 2013-04-03 10:45:58.793883 7fcdffcdb700 10 log_queue is 1 last_log 186 sent 185 num 1 unsent 1 sending 1 -7246> 2013-04-03 10:45:58.793885 7fcdffcdb700 10 will send 2013-04-03 10:45:58.793882 mon.0 10.252.24.25:6789/0 186 : [INF] pgmap v29120: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1696B/s wr, 0op/s -7245> 2013-04-03 10:45:58.793893 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24e8c00 -7244> 2013-04-03 10:45:58.793947 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos -7243> 2013-04-03 10:45:58.793967 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos version 394111 summary v 394111 -7242> 2013-04-03 10:45:58.793986 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7241> 2013-04-03 10:45:58.794008 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24e8c00 con 0x1bac160 -7240> 2013-04-03 10:45:58.794030 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos -7239> 2013-04-03 10:45:58.794046 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos version 394111 summary v 394111 -7238> 2013-04-03 10:45:58.794055 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7237> 2013-04-03 10:45:58.794066 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 preprocess_log log(1 entries) v1 from mon.0 -7236> 2013-04-03 10:45:58.794090 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7235> 2013-04-03 10:45:58.794102 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 prepare_log log(1 entries) v1 from mon.0 -7234> 2013-04-03 10:45:58.794113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 logging 2013-04-03 10:45:58.793882 mon.0 10.252.24.25:6789/0 186 : [INF] pgmap v29120: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1696B/s wr, 0op/s -7233> 2013-04-03 10:45:58.807305 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114029 ==== paxos(lease_ack lc 427519 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (710086262 0 0) 0x20f4a00 con 0x208a840 -7232> 2013-04-03 10:45:58.807387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos -7231> 2013-04-03 10:45:58.807407 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394111 update_from_paxos version 394111 summary v 394111 -7230> 2013-04-03 10:45:58.807424 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -7229> 2013-04-03 10:45:58.871470 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) -7228> 2013-04-03 10:45:58.871496 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader -7227> 2013-04-03 10:45:59.597843 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 189 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140473 v187) v2 ==== 289+0+0 (3259755305 0 0) 0x273adc0 con 0x2118420 -7226> 2013-04-03 10:45:59.597926 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140473 v187) v2 from mds.? 10.252.58.31:6800/10908 -7225> 2013-04-03 10:45:59.597939 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140473 v187) v2 -- ?+0 0x2cc5600 con 0x2118420 -7224> 2013-04-03 10:45:59.787251 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394111 encode_pending v394112 -7223> 2013-04-03 10:45:59.787279 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427519) queue_proposal bl 443 bytes; ctx = 0x2c3dbf0 -7222> 2013-04-03 10:45:59.787287 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427519) propose_queued 427520 443 bytes -7221> 2013-04-03 10:45:59.788414 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427519 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4780 -7220> 2013-04-03 10:45:59.832063 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114030 ==== paxos(accept lc 427519 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3538371682 0 0) 0x20f4780 con 0x208a840 -7219> 2013-04-03 10:45:59.832385 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427520 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4a00 -7218> 2013-04-03 10:45:59.832408 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427520 fc 427469 pn 0 opn 0) v3 -- ?+0 0x20f4280 -7217> 2013-04-03 10:45:59.832457 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos -7216> 2013-04-03 10:45:59.832493 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394111 -7215> 2013-04-03 10:45:59.832508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos latest full 393798 -7214> 2013-04-03 10:45:59.832526 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394112 update_from_paxos applying incremental log 394112 2013-04-03 10:45:58.793882 mon.0 10.252.24.25:6789/0 186 : [INF] pgmap v29120: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1696B/s wr, 0op/s -7213> 2013-04-03 10:45:59.832577 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 check_subs -7212> 2013-04-03 10:45:59.832612 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 create_pending v 394113 -7211> 2013-04-03 10:45:59.832634 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394112 _updated_log for mon.0 10.252.24.25:6789/0 -7210> 2013-04-03 10:45:59.832647 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 186) v1 -- ?+0 0x28a5e00 con 0x1bac160 -7209> 2013-04-03 10:45:59.832701 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos -7208> 2013-04-03 10:45:59.832717 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 -7207> 2013-04-03 10:45:59.832738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -7206> 2013-04-03 10:45:59.832761 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 186) v1 ==== 0+0+0 (0 0 0) 0x28a5e00 con 0x1bac160 -7205> 2013-04-03 10:45:59.832769 7fcdffcdb700 10 handle_log_ack log(last 186) v1 -7204> 2013-04-03 10:45:59.832770 7fcdffcdb700 10 logged 2013-04-03 10:45:58.793882 mon.0 10.252.24.25:6789/0 186 : [INF] pgmap v29120: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1696B/s wr, 0op/s -7203> 2013-04-03 10:45:59.857428 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114031 ==== paxos(lease_ack lc 427520 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (1781285752 0 0) 0x20f4280 con 0x208a840 -7202> 2013-04-03 10:45:59.857515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos -7201> 2013-04-03 10:45:59.857536 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 -7200> 2013-04-03 10:45:59.857554 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -7199> 2013-04-03 10:46:00.944619 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114032 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 188 v187) v2) to leader v1 ==== 560+0+0 (4207676743 0 0) 0x2cc42c0 con 0x208a840 -7198> 2013-04-03 10:46:00.944640 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -7197> 2013-04-03 10:46:00.944650 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4840 from 10.252.24.88:6789/0 -7196> 2013-04-03 10:46:00.944705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 188 v187) v2 from mds.0 10.252.24.25:6802/3144 -7195> 2013-04-03 10:46:00.944716 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 188 v187) v2 tid 17744) v2 -- ?+0 0x24f8000 con 0x208a840 -7194> 2013-04-03 10:46:01.395823 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 189 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21844 v187) v2 ==== 289+0+0 (3808477195 0 0) 0x2f7a8c0 con 0x200d420 -7193> 2013-04-03 10:46:01.395908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21844 v187) v2 from mds.? 10.252.24.88:6800/17213 -7192> 2013-04-03 10:46:01.395922 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21844 v187) v2 -- ?+0 0x2cc42c0 con 0x200d420 -7191> 2013-04-03 10:46:02.832475 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427520 fc 427469 pn 0 opn 0) v3 -- ?+0 0x20f4500 -7190> 2013-04-03 10:46:02.833167 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114033 ==== paxos(lease_ack lc 427520 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (57229997 0 0) 0x20f4500 con 0x208a840 -7189> 2013-04-03 10:46:02.833244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos -7188> 2013-04-03 10:46:02.833264 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 -7187> 2013-04-03 10:46:02.833284 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -7186> 2013-04-03 10:46:03.377319 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29120 check_down_pgs -7185> 2013-04-03 10:46:03.377353 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29120 v29120: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1696B/s wr, 0op/s -7184> 2013-04-03 10:46:03.377382 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -7183> 2013-04-03 10:46:03.377570 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -7182> 2013-04-03 10:46:03.377612 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -7181> 2013-04-03 10:46:03.377644 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos -7180> 2013-04-03 10:46:03.377663 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 -7179> 2013-04-03 10:46:03.377672 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394112 log -7178> 2013-04-03 10:46:03.377695 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -7177> 2013-04-03 10:46:03.377715 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -7176> 2013-04-03 10:46:03.597980 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 190 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140474 v187) v2 ==== 289+0+0 (2818058926 0 0) 0x273b600 con 0x2118420 -7175> 2013-04-03 10:46:03.598051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140474 v187) v2 from mds.? 10.252.58.31:6800/10908 -7174> 2013-04-03 10:46:03.598064 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140474 v187) v2 -- ?+0 0x2f7a8c0 con 0x2118420 -7173> 2013-04-03 10:46:04.944732 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114034 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 189 v187) v2) to leader v1 ==== 560+0+0 (4085633860 0 0) 0x273adc0 con 0x208a840 -7172> 2013-04-03 10:46:04.944754 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -7171> 2013-04-03 10:46:04.944764 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc5340 from 10.252.24.88:6789/0 -7170> 2013-04-03 10:46:04.944818 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 189 v187) v2 from mds.0 10.252.24.25:6802/3144 -7169> 2013-04-03 10:46:04.944828 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 189 v187) v2 tid 17745) v2 -- ?+0 0x24f8b40 con 0x208a840 -7168> 2013-04-03 10:46:05.395908 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 190 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21845 v187) v2 ==== 289+0+0 (3228467062 0 0) 0x20e4080 con 0x200d420 -7167> 2013-04-03 10:46:05.395994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21845 v187) v2 from mds.? 10.252.24.88:6800/17213 -7166> 2013-04-03 10:46:05.396007 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21845 v187) v2 -- ?+0 0x273adc0 con 0x200d420 -7165> 2013-04-03 10:46:05.832548 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427520 fc 427469 pn 0 opn 0) v3 -- ?+0 0x29b4c80 -7164> 2013-04-03 10:46:05.833339 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114035 ==== paxos(lease_ack lc 427520 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (2062284291 0 0) 0x29b4c80 con 0x208a840 -7163> 2013-04-03 10:46:05.833428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos -7162> 2013-04-03 10:46:05.833449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 -7161> 2013-04-03 10:46:05.833468 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -7160> 2013-04-03 10:46:07.598113 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 191 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140475 v187) v2 ==== 289+0+0 (2224461267 0 0) 0x273a580 con 0x2118420 -7159> 2013-04-03 10:46:07.598196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140475 v187) v2 from mds.? 10.252.58.31:6800/10908 -7158> 2013-04-03 10:46:07.598210 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140475 v187) v2 -- ?+0 0x20e4080 con 0x2118420 -7157> 2013-04-03 10:46:08.377845 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29120 v29120: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1696B/s wr, 0op/s -7156> 2013-04-03 10:46:08.377888 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -7155> 2013-04-03 10:46:08.377913 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -7154> 2013-04-03 10:46:08.377960 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -7153> 2013-04-03 10:46:08.377992 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos -7152> 2013-04-03 10:46:08.378010 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 -7151> 2013-04-03 10:46:08.378019 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394112 log -7150> 2013-04-03 10:46:08.378047 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -7149> 2013-04-03 10:46:08.378067 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -7148> 2013-04-03 10:46:08.832612 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427520 fc 427469 pn 0 opn 0) v3 -- ?+0 0x29b4780 -7147> 2013-04-03 10:46:08.833383 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114036 ==== paxos(lease_ack lc 427520 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3254278032 0 0) 0x29b4780 con 0x208a840 -7146> 2013-04-03 10:46:08.833460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos -7145> 2013-04-03 10:46:08.833481 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 -7144> 2013-04-03 10:46:08.833498 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -7143> 2013-04-03 10:46:08.944949 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114037 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 190 v187) v2) to leader v1 ==== 560+0+0 (3898585409 0 0) 0x273b600 con 0x208a840 -7142> 2013-04-03 10:46:08.944971 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -7141> 2013-04-03 10:46:08.944981 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc5b80 from 10.252.24.88:6789/0 -7140> 2013-04-03 10:46:08.945034 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 190 v187) v2 from mds.0 10.252.24.25:6802/3144 -7139> 2013-04-03 10:46:08.945045 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 190 v187) v2 tid 17746) v2 -- ?+0 0x24f8000 con 0x208a840 -7138> 2013-04-03 10:46:09.395993 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 191 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21846 v187) v2 ==== 289+0+0 (2782772977 0 0) 0x20e48c0 con 0x200d420 -7137> 2013-04-03 10:46:09.396065 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21846 v187) v2 from mds.? 10.252.24.88:6800/17213 -7136> 2013-04-03 10:46:09.396077 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21846 v187) v2 -- ?+0 0x273b600 con 0x200d420 -7135> 2013-04-03 10:46:11.225290 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2fe4780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33172/0 -7134> 2013-04-03 10:46:11.225333 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2fe4780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33172/0) -7133> 2013-04-03 10:46:11.225348 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2fe4780 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -7132> 2013-04-03 10:46:11.225538 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2fe4780 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -7131> 2013-04-03 10:46:11.598245 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 192 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140476 v187) v2 ==== 289+0+0 (1822402976 0 0) 0x273a2c0 con 0x2118420 -7130> 2013-04-03 10:46:11.598328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140476 v187) v2 from mds.? 10.252.58.31:6800/10908 -7129> 2013-04-03 10:46:11.598343 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140476 v187) v2 -- ?+0 0x20e48c0 con 0x2118420 -7128> 2013-04-03 10:46:11.634585 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114038 ==== forward(pg_stats(1 pgs tid 59 v 605) v1) to leader v1 ==== 753+0+0 (1723743309 0 0) 0x273a580 con 0x208a840 -7127> 2013-04-03 10:46:11.634599 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -7126> 2013-04-03 10:46:11.634607 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 -7125> 2013-04-03 10:46:11.634644 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29120 preprocess_query pg_stats(1 pgs tid 59 v 605) v1 from osd.1 10.252.24.25:6805/3572 -7124> 2013-04-03 10:46:11.634670 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29120 prepare_update pg_stats(1 pgs tid 59 v 605) v1 from osd.1 10.252.24.25:6805/3572 -7123> 2013-04-03 10:46:11.634676 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29120 prepare_pg_stats pg_stats(1 pgs tid 59 v 605) v1 from osd.1 -7122> 2013-04-03 10:46:11.634682 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29120 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -7121> 2013-04-03 10:46:11.684864 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29120 encode_pending v 29121 -7120> 2013-04-03 10:46:11.684913 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427520) queue_proposal bl 650 bytes; ctx = 0x2c3dbd0 -7119> 2013-04-03 10:46:11.684923 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427520) propose_queued 427521 650 bytes -7118> 2013-04-03 10:46:11.685878 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427520 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4000 -7117> 2013-04-03 10:46:11.722892 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114039 ==== paxos(accept lc 427520 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (610907054 0 0) 0x29b4000 con 0x208a840 -7116> 2013-04-03 10:46:11.723267 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427521 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2a00 -7115> 2013-04-03 10:46:11.723292 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427521 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2df2780 -7114> 2013-04-03 10:46:11.723382 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29120 update_from_paxos applying incremental 29121 -7113> 2013-04-03 10:46:11.723402 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 v29121: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -7112> 2013-04-03 10:46:11.723421 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 send_pg_creates to 0 pgs -7111> 2013-04-03 10:46:11.723423 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 update_logger -7110> 2013-04-03 10:46:11.723433 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 create_pending v 29122 -7109> 2013-04-03 10:46:11.723442 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29121 _updated_stats for osd.1 10.252.24.25:6805/3572 -7108> 2013-04-03 10:46:11.723450 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 59) v1 tid 17747) v2 -- ?+0 0x24f8b40 con 0x208a840 -7107> 2013-04-03 10:46:11.723460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 check_osd_map already seen 605 >= 605 -7106> 2013-04-03 10:46:11.723463 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 update_logger -7105> 2013-04-03 10:46:11.723476 7fcdffcdb700 0 log [INF] : pgmap v29121: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -7104> 2013-04-03 10:46:11.723477 7fcdffcdb700 10 send_log to self -7103> 2013-04-03 10:46:11.723478 7fcdffcdb700 10 log_queue is 1 last_log 187 sent 186 num 1 unsent 1 sending 1 -7102> 2013-04-03 10:46:11.723480 7fcdffcdb700 10 will send 2013-04-03 10:46:11.723477 mon.0 10.252.24.25:6789/0 187 : [INF] pgmap v29121: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -7101> 2013-04-03 10:46:11.723488 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24e8c00 -7100> 2013-04-03 10:46:11.723542 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos -7099> 2013-04-03 10:46:11.723561 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 -7098> 2013-04-03 10:46:11.723579 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -7097> 2013-04-03 10:46:11.723600 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24e8c00 con 0x1bac160 -7096> 2013-04-03 10:46:11.723622 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos -7095> 2013-04-03 10:46:11.723637 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 -7094> 2013-04-03 10:46:11.723646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7093> 2013-04-03 10:46:11.723658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 preprocess_log log(1 entries) v1 from mon.0 -7092> 2013-04-03 10:46:11.723682 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7091> 2013-04-03 10:46:11.723694 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 prepare_log log(1 entries) v1 from mon.0 -7090> 2013-04-03 10:46:11.723704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 logging 2013-04-03 10:46:11.723477 mon.0 10.252.24.25:6789/0 187 : [INF] pgmap v29121: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -7089> 2013-04-03 10:46:11.736227 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114040 ==== paxos(lease_ack lc 427521 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (40714603 0 0) 0x2df2780 con 0x208a840 -7088> 2013-04-03 10:46:11.736324 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos -7087> 2013-04-03 10:46:11.736345 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394112 update_from_paxos version 394112 summary v 394112 -7086> 2013-04-03 10:46:11.736365 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -7085> 2013-04-03 10:46:11.800894 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114041 ==== forward(pg_stats(1 pgs tid 30461 v 605) v1) to leader v1 ==== 757+0+0 (648258486 0 0) 0x2cc4dc0 con 0x208a840 -7084> 2013-04-03 10:46:11.800924 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -7083> 2013-04-03 10:46:11.800935 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -7082> 2013-04-03 10:46:11.800989 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 preprocess_query pg_stats(1 pgs tid 30461 v 605) v1 from osd.3 10.252.24.88:6801/17538 -7081> 2013-04-03 10:46:11.801014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 prepare_update pg_stats(1 pgs tid 30461 v 605) v1 from osd.3 10.252.24.88:6801/17538 -7080> 2013-04-03 10:46:11.801019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 prepare_pg_stats pg_stats(1 pgs tid 30461 v 605) v1 from osd.3 -7079> 2013-04-03 10:46:11.801026 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29121 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -7078> 2013-04-03 10:46:12.723097 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394112 encode_pending v394113 -7077> 2013-04-03 10:46:12.723127 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427521) queue_proposal bl 443 bytes; ctx = 0x2c3dd00 -7076> 2013-04-03 10:46:12.723134 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427521) propose_queued 427522 443 bytes -7075> 2013-04-03 10:46:12.724116 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427521 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2280 -7074> 2013-04-03 10:46:12.724202 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29121 encode_pending v 29122 -7073> 2013-04-03 10:46:12.724524 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427469..427521) queue_proposal bl 654 bytes; ctx = 0x2c3dce0 -7072> 2013-04-03 10:46:12.724534 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427469..427521) propose_new_value not active; proposal queued -7071> 2013-04-03 10:46:12.892520 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114042 ==== paxos(accept lc 427521 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4070729135 0 0) 0x2df2280 con 0x208a840 -7070> 2013-04-03 10:46:12.893695 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427522 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2780 -7069> 2013-04-03 10:46:12.893715 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427522 fc 427469 pn 0 opn 0) v3 -- ?+0 0x29b4000 -7068> 2013-04-03 10:46:12.893754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos -7067> 2013-04-03 10:46:12.893791 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos version 394113 summary v 394112 -7066> 2013-04-03 10:46:12.893805 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos latest full 393798 -7065> 2013-04-03 10:46:12.893823 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394113 update_from_paxos applying incremental log 394113 2013-04-03 10:46:11.723477 mon.0 10.252.24.25:6789/0 187 : [INF] pgmap v29121: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -7064> 2013-04-03 10:46:12.893876 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 check_subs -7063> 2013-04-03 10:46:12.893908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 create_pending v 394114 -7062> 2013-04-03 10:46:12.893931 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394113 _updated_log for mon.0 10.252.24.25:6789/0 -7061> 2013-04-03 10:46:12.893944 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 187) v1 -- ?+0 0x28a5a80 con 0x1bac160 -7060> 2013-04-03 10:46:12.893972 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427522) propose_queued 427523 654 bytes -7059> 2013-04-03 10:46:12.894199 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427522 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2c80 -7058> 2013-04-03 10:46:12.894219 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 187) v1 ==== 0+0+0 (0 0 0) 0x28a5a80 con 0x1bac160 -7057> 2013-04-03 10:46:12.894226 7fcdffcdb700 10 handle_log_ack log(last 187) v1 -7056> 2013-04-03 10:46:12.894229 7fcdffcdb700 10 logged 2013-04-03 10:46:11.723477 mon.0 10.252.24.25:6789/0 187 : [INF] pgmap v29121: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -7055> 2013-04-03 10:46:12.905835 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114043 ==== paxos(lease_ack lc 427522 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3012450245 0 0) 0x2df2c80 con 0x208a840 -7054> 2013-04-03 10:46:12.931104 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114044 ==== paxos(accept lc 427522 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2434335424 0 0) 0x29b4000 con 0x208a840 -7053> 2013-04-03 10:46:12.931478 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427523 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2c80 -7052> 2013-04-03 10:46:12.931503 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427523 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2df2280 -7051> 2013-04-03 10:46:12.931594 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29121 update_from_paxos applying incremental 29122 -7050> 2013-04-03 10:46:12.931615 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 v29122: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 366B/s wr, 0op/s -7049> 2013-04-03 10:46:12.931634 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 send_pg_creates to 0 pgs -7048> 2013-04-03 10:46:12.931637 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 update_logger -7047> 2013-04-03 10:46:12.931647 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 create_pending v 29123 -7046> 2013-04-03 10:46:12.931660 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29122 _updated_stats for osd.3 10.252.24.88:6801/17538 -7045> 2013-04-03 10:46:12.931667 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30461) v1 tid 17748) v2 -- ?+0 0x24f8000 con 0x208a840 -7044> 2013-04-03 10:46:12.931679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 check_osd_map already seen 605 >= 605 -7043> 2013-04-03 10:46:12.931682 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 update_logger -7042> 2013-04-03 10:46:12.931692 7fcdffcdb700 0 log [INF] : pgmap v29122: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 366B/s wr, 0op/s -7041> 2013-04-03 10:46:12.931702 7fcdffcdb700 10 send_log to self -7040> 2013-04-03 10:46:12.931703 7fcdffcdb700 10 log_queue is 1 last_log 188 sent 187 num 1 unsent 1 sending 1 -7039> 2013-04-03 10:46:12.931705 7fcdffcdb700 10 will send 2013-04-03 10:46:12.931701 mon.0 10.252.24.25:6789/0 188 : [INF] pgmap v29122: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 366B/s wr, 0op/s -7038> 2013-04-03 10:46:12.931712 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b76a00 -7037> 2013-04-03 10:46:12.931767 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos -7036> 2013-04-03 10:46:12.931787 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos version 394113 summary v 394113 -7035> 2013-04-03 10:46:12.931806 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -7034> 2013-04-03 10:46:12.931827 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76a00 con 0x1bac160 -7033> 2013-04-03 10:46:12.931849 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos -7032> 2013-04-03 10:46:12.931863 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos version 394113 summary v 394113 -7031> 2013-04-03 10:46:12.931872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7030> 2013-04-03 10:46:12.931884 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 preprocess_log log(1 entries) v1 from mon.0 -7029> 2013-04-03 10:46:12.931909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -7028> 2013-04-03 10:46:12.931920 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 prepare_log log(1 entries) v1 from mon.0 -7027> 2013-04-03 10:46:12.931931 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 logging 2013-04-03 10:46:12.931701 mon.0 10.252.24.25:6789/0 188 : [INF] pgmap v29122: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 366B/s wr, 0op/s -7026> 2013-04-03 10:46:12.943039 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114045 ==== paxos(lease_ack lc 427523 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (2651086292 0 0) 0x2df2280 con 0x208a840 -7025> 2013-04-03 10:46:12.943074 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos -7024> 2013-04-03 10:46:12.943090 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos version 394113 summary v 394113 -7023> 2013-04-03 10:46:12.943105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -7022> 2013-04-03 10:46:12.944749 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114046 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 191 v187) v2) to leader v1 ==== 560+0+0 (1283698463 0 0) 0x2a84340 con 0x208a840 -7021> 2013-04-03 10:46:12.944766 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -7020> 2013-04-03 10:46:12.944775 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83580 from 10.252.24.88:6789/0 -7019> 2013-04-03 10:46:12.944812 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 191 v187) v2 from mds.0 10.252.24.25:6802/3144 -7018> 2013-04-03 10:46:12.944823 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 191 v187) v2 tid 17749) v2 -- ?+0 0x24f8b40 con 0x208a840 -7017> 2013-04-03 10:46:13.281339 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114047 ==== forward(pg_stats(1 pgs tid 41 v 605) v1) to leader v1 ==== 757+0+0 (181989828 0 0) 0x2cc4dc0 con 0x208a840 -7016> 2013-04-03 10:46:13.281367 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -7015> 2013-04-03 10:46:13.281377 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -7014> 2013-04-03 10:46:13.281428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 preprocess_query pg_stats(1 pgs tid 41 v 605) v1 from osd.2 10.252.24.30:6801/5656 -7013> 2013-04-03 10:46:13.281452 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 prepare_update pg_stats(1 pgs tid 41 v 605) v1 from osd.2 10.252.24.30:6801/5656 -7012> 2013-04-03 10:46:13.281458 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 prepare_pg_stats pg_stats(1 pgs tid 41 v 605) v1 from osd.2 -7011> 2013-04-03 10:46:13.281464 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29122 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -7010> 2013-04-03 10:46:13.378190 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29122 check_down_pgs -7009> 2013-04-03 10:46:13.378222 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29122 v29122: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 366B/s wr, 0op/s -7008> 2013-04-03 10:46:13.378252 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -7007> 2013-04-03 10:46:13.378281 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -7006> 2013-04-03 10:46:13.378319 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -7005> 2013-04-03 10:46:13.378354 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos -7004> 2013-04-03 10:46:13.378374 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394113 update_from_paxos version 394113 summary v 394113 -7003> 2013-04-03 10:46:13.378384 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394113 log -7002> 2013-04-03 10:46:13.378402 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -7001> 2013-04-03 10:46:13.378422 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -7000> 2013-04-03 10:46:13.396176 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 192 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21847 v187) v2 ==== 289+0+0 (2259878284 0 0) 0x20e4b80 con 0x200d420 -6999> 2013-04-03 10:46:13.396257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21847 v187) v2 from mds.? 10.252.24.88:6800/17213 -6998> 2013-04-03 10:46:13.396270 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21847 v187) v2 -- ?+0 0x2a83dc0 con 0x200d420 -6997> 2013-04-03 10:46:13.931308 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29122 encode_pending v 29123 -6996> 2013-04-03 10:46:13.931346 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427523) queue_proposal bl 654 bytes; ctx = 0x2c3dbe0 -6995> 2013-04-03 10:46:13.931354 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427523) propose_queued 427524 654 bytes -6994> 2013-04-03 10:46:13.932500 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427523 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2500 -6993> 2013-04-03 10:46:13.932592 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394113 encode_pending v394114 -6992> 2013-04-03 10:46:13.932619 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427469..427523) queue_proposal bl 442 bytes; ctx = 0x2c3dbc0 -6991> 2013-04-03 10:46:13.932624 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427469..427523) propose_new_value not active; proposal queued -6990> 2013-04-03 10:46:14.050872 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114048 ==== paxos(accept lc 427523 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3285210879 0 0) 0x2df2500 con 0x208a840 -6989> 2013-04-03 10:46:14.051818 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427524 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2280 -6988> 2013-04-03 10:46:14.051839 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427524 fc 427469 pn 0 opn 0) v3 -- ?+0 0x29b4000 -6987> 2013-04-03 10:46:14.051943 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29122 update_from_paxos applying incremental 29123 -6986> 2013-04-03 10:46:14.051968 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 v29123: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 5925B/s wr, 0op/s -6985> 2013-04-03 10:46:14.051987 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 send_pg_creates to 0 pgs -6984> 2013-04-03 10:46:14.051990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 update_logger -6983> 2013-04-03 10:46:14.052000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 create_pending v 29124 -6982> 2013-04-03 10:46:14.052009 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29123 _updated_stats for osd.2 10.252.24.30:6801/5656 -6981> 2013-04-03 10:46:14.052016 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 41) v1 tid 17750) v2 -- ?+0 0x24f8000 con 0x208a840 -6980> 2013-04-03 10:46:14.052027 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 check_osd_map already seen 605 >= 605 -6979> 2013-04-03 10:46:14.052030 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 update_logger -6978> 2013-04-03 10:46:14.052040 7fcdffcdb700 0 log [INF] : pgmap v29123: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 5925B/s wr, 0op/s -6977> 2013-04-03 10:46:14.052042 7fcdffcdb700 10 send_log to self -6976> 2013-04-03 10:46:14.052043 7fcdffcdb700 10 log_queue is 2 last_log 189 sent 188 num 2 unsent 1 sending 1 -6975> 2013-04-03 10:46:14.052045 7fcdffcdb700 10 will send 2013-04-03 10:46:12.931701 mon.0 10.252.24.25:6789/0 188 : [INF] pgmap v29122: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 366B/s wr, 0op/s -6974> 2013-04-03 10:46:14.052052 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b76c00 -6973> 2013-04-03 10:46:14.052082 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427524) propose_queued 427525 442 bytes -6972> 2013-04-03 10:46:14.052279 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427524 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4780 -6971> 2013-04-03 10:46:14.052298 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76c00 con 0x1bac160 -6970> 2013-04-03 10:46:14.076094 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114049 ==== paxos(lease_ack lc 427524 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (649414533 0 0) 0x29b4780 con 0x208a840 -6969> 2013-04-03 10:46:14.087926 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114050 ==== paxos(accept lc 427524 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3160946933 0 0) 0x29b4000 con 0x208a840 -6968> 2013-04-03 10:46:14.089037 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427525 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4780 -6967> 2013-04-03 10:46:14.089062 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427525 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2df2500 -6966> 2013-04-03 10:46:14.089107 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos -6965> 2013-04-03 10:46:14.089144 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394113 -6964> 2013-04-03 10:46:14.089159 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos latest full 393798 -6963> 2013-04-03 10:46:14.089177 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394114 update_from_paxos applying incremental log 394114 2013-04-03 10:46:12.931701 mon.0 10.252.24.25:6789/0 188 : [INF] pgmap v29122: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 366B/s wr, 0op/s -6962> 2013-04-03 10:46:14.089227 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 check_subs -6961> 2013-04-03 10:46:14.089259 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 create_pending v 394115 -6960> 2013-04-03 10:46:14.089282 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394114 _updated_log for mon.0 10.252.24.25:6789/0 -6959> 2013-04-03 10:46:14.089294 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 188) v1 -- ?+0 0x2e8b1c0 con 0x1bac160 -6958> 2013-04-03 10:46:14.089324 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos -6957> 2013-04-03 10:46:14.089339 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 -6956> 2013-04-03 10:46:14.089348 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6955> 2013-04-03 10:46:14.089360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 preprocess_log log(1 entries) v1 from mon.0 -6954> 2013-04-03 10:46:14.089378 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 nothing new -6953> 2013-04-03 10:46:14.089433 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos -6952> 2013-04-03 10:46:14.089449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 -6951> 2013-04-03 10:46:14.089467 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6950> 2013-04-03 10:46:14.089489 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 188) v1 ==== 0+0+0 (0 0 0) 0x2e8b1c0 con 0x1bac160 -6949> 2013-04-03 10:46:14.089497 7fcdffcdb700 10 handle_log_ack log(last 188) v1 -6948> 2013-04-03 10:46:14.089500 7fcdffcdb700 10 logged 2013-04-03 10:46:12.931701 mon.0 10.252.24.25:6789/0 188 : [INF] pgmap v29122: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 366B/s wr, 0op/s -6947> 2013-04-03 10:46:14.101366 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114051 ==== paxos(lease_ack lc 427525 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (1638713861 0 0) 0x2df2500 con 0x208a840 -6946> 2013-04-03 10:46:14.101412 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos -6945> 2013-04-03 10:46:14.101429 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 -6944> 2013-04-03 10:46:14.101444 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6943> 2013-04-03 10:46:15.598334 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 193 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140477 v187) v2 ==== 289+0+0 (1341200093 0 0) 0x273b340 con 0x2118420 -6942> 2013-04-03 10:46:15.598414 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140477 v187) v2 from mds.? 10.252.58.31:6800/10908 -6941> 2013-04-03 10:46:15.598428 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140477 v187) v2 -- ?+0 0x20e4b80 con 0x2118420 -6940> 2013-04-03 10:46:16.945192 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114052 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 192 v187) v2) to leader v1 ==== 560+0+0 (3759462058 0 0) 0x2a83000 con 0x208a840 -6939> 2013-04-03 10:46:16.945212 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -6938> 2013-04-03 10:46:16.945222 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a832c0 from 10.252.24.88:6789/0 -6937> 2013-04-03 10:46:16.945278 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 192 v187) v2 from mds.0 10.252.24.25:6802/3144 -6936> 2013-04-03 10:46:16.945289 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 192 v187) v2 tid 17751) v2 -- ?+0 0x24f8b40 con 0x208a840 -6935> 2013-04-03 10:46:17.089135 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427525 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2df2000 -6934> 2013-04-03 10:46:17.089878 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114053 ==== paxos(lease_ack lc 427525 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (2435674043 0 0) 0x2df2000 con 0x208a840 -6933> 2013-04-03 10:46:17.089954 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos -6932> 2013-04-03 10:46:17.089975 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 -6931> 2013-04-03 10:46:17.089993 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6930> 2013-04-03 10:46:17.396290 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 193 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21848 v187) v2 ==== 289+0+0 (1881387238 0 0) 0x2efa080 con 0x200d420 -6929> 2013-04-03 10:46:17.396370 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21848 v187) v2 from mds.? 10.252.24.88:6800/17213 -6928> 2013-04-03 10:46:17.396387 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21848 v187) v2 -- ?+0 0x2a83000 con 0x200d420 -6927> 2013-04-03 10:46:17.421093 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 194 ==== mon_subscribe({mdsmap=188+,monmap=6+}) v2 ==== 42+0+0 (435898880 0 0) 0x2e8b000 con 0x200d420 -6926> 2013-04-03 10:46:17.421121 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 -6925> 2013-04-03 10:46:17.421132 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -6924> 2013-04-03 10:46:17.421135 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/17213 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8b1c0 -6923> 2013-04-03 10:46:18.281973 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114054 ==== forward(pg_stats(1 pgs tid 42 v 605) v1) to leader v1 ==== 757+0+0 (1658576591 0 0) 0x273b340 con 0x208a840 -6922> 2013-04-03 10:46:18.282001 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -6921> 2013-04-03 10:46:18.282011 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -6920> 2013-04-03 10:46:18.282064 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 preprocess_query pg_stats(1 pgs tid 42 v 605) v1 from osd.2 10.252.24.30:6801/5656 -6919> 2013-04-03 10:46:18.282088 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 prepare_update pg_stats(1 pgs tid 42 v 605) v1 from osd.2 10.252.24.30:6801/5656 -6918> 2013-04-03 10:46:18.282094 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 prepare_pg_stats pg_stats(1 pgs tid 42 v 605) v1 from osd.2 -6917> 2013-04-03 10:46:18.282100 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29123 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -6916> 2013-04-03 10:46:18.332284 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29123 encode_pending v 29124 -6915> 2013-04-03 10:46:18.332323 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427525) queue_proposal bl 654 bytes; ctx = 0x2c3d840 -6914> 2013-04-03 10:46:18.332333 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427525) propose_queued 427526 654 bytes -6913> 2013-04-03 10:46:18.333523 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427525 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3222c80 -6912> 2013-04-03 10:46:18.378553 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -6911> 2013-04-03 10:46:18.378593 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -6910> 2013-04-03 10:46:18.378622 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos -6909> 2013-04-03 10:46:18.378641 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 -6908> 2013-04-03 10:46:18.378650 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394114 log -6907> 2013-04-03 10:46:18.378668 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6906> 2013-04-03 10:46:18.378687 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -6905> 2013-04-03 10:46:18.464938 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114055 ==== paxos(accept lc 427525 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1611153159 0 0) 0x3222c80 con 0x208a840 -6904> 2013-04-03 10:46:18.465275 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427526 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2000 -6903> 2013-04-03 10:46:18.465298 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427526 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2df2500 -6902> 2013-04-03 10:46:18.465397 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29123 update_from_paxos applying incremental 29124 -6901> 2013-04-03 10:46:18.465419 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 v29124: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2921B/s wr, 0op/s -6900> 2013-04-03 10:46:18.465437 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 send_pg_creates to 0 pgs -6899> 2013-04-03 10:46:18.465440 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 update_logger -6898> 2013-04-03 10:46:18.465453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 create_pending v 29125 -6897> 2013-04-03 10:46:18.465464 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29124 _updated_stats for osd.2 10.252.24.30:6801/5656 -6896> 2013-04-03 10:46:18.465472 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 42) v1 tid 17752) v2 -- ?+0 0x24f8000 con 0x208a840 -6895> 2013-04-03 10:46:18.465482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 check_osd_map already seen 605 >= 605 -6894> 2013-04-03 10:46:18.465485 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 update_logger -6893> 2013-04-03 10:46:18.465495 7fcdffcdb700 0 log [INF] : pgmap v29124: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2921B/s wr, 0op/s -6892> 2013-04-03 10:46:18.465497 7fcdffcdb700 10 send_log to self -6891> 2013-04-03 10:46:18.465498 7fcdffcdb700 10 log_queue is 2 last_log 190 sent 188 num 2 unsent 2 sending 2 -6890> 2013-04-03 10:46:18.465499 7fcdffcdb700 10 will send 2013-04-03 10:46:14.052042 mon.0 10.252.24.25:6789/0 189 : [INF] pgmap v29123: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 5925B/s wr, 0op/s -6889> 2013-04-03 10:46:18.465507 7fcdffcdb700 10 will send 2013-04-03 10:46:18.465496 mon.0 10.252.24.25:6789/0 190 : [INF] pgmap v29124: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2921B/s wr, 0op/s -6888> 2013-04-03 10:46:18.465512 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b76c00 -6887> 2013-04-03 10:46:18.465564 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos -6886> 2013-04-03 10:46:18.465583 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 -6885> 2013-04-03 10:46:18.465601 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6884> 2013-04-03 10:46:18.465624 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76c00 con 0x1bac160 -6883> 2013-04-03 10:46:18.465646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos -6882> 2013-04-03 10:46:18.465661 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 -6881> 2013-04-03 10:46:18.465670 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -6880> 2013-04-03 10:46:18.465681 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 preprocess_log log(2 entries) v1 from mon.0 -6879> 2013-04-03 10:46:18.465714 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -6878> 2013-04-03 10:46:18.465727 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 prepare_log log(2 entries) v1 from mon.0 -6877> 2013-04-03 10:46:18.465737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 logging 2013-04-03 10:46:14.052042 mon.0 10.252.24.25:6789/0 189 : [INF] pgmap v29123: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 5925B/s wr, 0op/s -6876> 2013-04-03 10:46:18.465758 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 logging 2013-04-03 10:46:18.465496 mon.0 10.252.24.25:6789/0 190 : [INF] pgmap v29124: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2921B/s wr, 0op/s -6875> 2013-04-03 10:46:18.476924 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114056 ==== paxos(lease_ack lc 427526 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3049038744 0 0) 0x2df2500 con 0x208a840 -6874> 2013-04-03 10:46:18.477010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos -6873> 2013-04-03 10:46:18.477030 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394114 update_from_paxos version 394114 summary v 394114 -6872> 2013-04-03 10:46:18.477047 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6871> 2013-04-03 10:46:19.465136 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394114 encode_pending v394115 -6870> 2013-04-03 10:46:19.465166 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427526) queue_proposal bl 726 bytes; ctx = 0x2c3dc00 -6869> 2013-04-03 10:46:19.465174 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427526) propose_queued 427527 726 bytes -6868> 2013-04-03 10:46:19.466327 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427526 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3222a00 -6867> 2013-04-03 10:46:19.493337 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114057 ==== paxos(accept lc 427526 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2912243186 0 0) 0x3222a00 con 0x208a840 -6866> 2013-04-03 10:46:19.493704 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427527 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2df2500 -6865> 2013-04-03 10:46:19.493725 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427527 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3222c80 -6864> 2013-04-03 10:46:19.493775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos -6863> 2013-04-03 10:46:19.493827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos version 394115 summary v 394114 -6862> 2013-04-03 10:46:19.493842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos latest full 393798 -6861> 2013-04-03 10:46:19.493860 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394115 update_from_paxos applying incremental log 394115 2013-04-03 10:46:14.052042 mon.0 10.252.24.25:6789/0 189 : [INF] pgmap v29123: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 5925B/s wr, 0op/s -6860> 2013-04-03 10:46:19.493889 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394115 update_from_paxos applying incremental log 394115 2013-04-03 10:46:18.465496 mon.0 10.252.24.25:6789/0 190 : [INF] pgmap v29124: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2921B/s wr, 0op/s -6859> 2013-04-03 10:46:19.493947 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 check_subs -6858> 2013-04-03 10:46:19.493981 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 create_pending v 394116 -6857> 2013-04-03 10:46:19.494003 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394115 _updated_log for mon.0 10.252.24.25:6789/0 -6856> 2013-04-03 10:46:19.494018 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 190) v1 -- ?+0 0x2e8b700 con 0x1bac160 -6855> 2013-04-03 10:46:19.494071 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos -6854> 2013-04-03 10:46:19.494086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos version 394115 summary v 394115 -6853> 2013-04-03 10:46:19.494104 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6852> 2013-04-03 10:46:19.494127 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 190) v1 ==== 0+0+0 (0 0 0) 0x2e8b700 con 0x1bac160 -6851> 2013-04-03 10:46:19.494135 7fcdffcdb700 10 handle_log_ack log(last 190) v1 -6850> 2013-04-03 10:46:19.494138 7fcdffcdb700 10 logged 2013-04-03 10:46:14.052042 mon.0 10.252.24.25:6789/0 189 : [INF] pgmap v29123: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 5925B/s wr, 0op/s -6849> 2013-04-03 10:46:19.494144 7fcdffcdb700 10 logged 2013-04-03 10:46:18.465496 mon.0 10.252.24.25:6789/0 190 : [INF] pgmap v29124: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2921B/s wr, 0op/s -6848> 2013-04-03 10:46:19.505351 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114058 ==== paxos(lease_ack lc 427527 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (238165113 0 0) 0x3222c80 con 0x208a840 -6847> 2013-04-03 10:46:19.505433 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos -6846> 2013-04-03 10:46:19.505454 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos version 394115 summary v 394115 -6845> 2013-04-03 10:46:19.505472 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6844> 2013-04-03 10:46:19.598451 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 194 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140478 v187) v2 ==== 289+0+0 (709023578 0 0) 0x273bb80 con 0x2118420 -6843> 2013-04-03 10:46:19.598526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140478 v187) v2 from mds.? 10.252.58.31:6800/10908 -6842> 2013-04-03 10:46:19.598539 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140478 v187) v2 -- ?+0 0x273b340 con 0x2118420 -6841> 2013-04-03 10:46:20.945224 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114059 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 193 v187) v2) to leader v1 ==== 560+0+0 (1671833012 0 0) 0x2a84600 con 0x208a840 -6840> 2013-04-03 10:46:20.945247 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -6839> 2013-04-03 10:46:20.945258 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83840 from 10.252.24.88:6789/0 -6838> 2013-04-03 10:46:20.945316 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 193 v187) v2 from mds.0 10.252.24.25:6802/3144 -6837> 2013-04-03 10:46:20.945329 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 193 v187) v2 tid 17753) v2 -- ?+0 0x24f8d80 con 0x208a840 -6836> 2013-04-03 10:46:21.241224 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2fe4c80 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33173/0 -6835> 2013-04-03 10:46:21.241264 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2fe4c80 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33173/0) -6834> 2013-04-03 10:46:21.241279 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2fe4c80 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -6833> 2013-04-03 10:46:21.241463 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2fe4c80 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -6832> 2013-04-03 10:46:21.396409 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 195 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21849 v187) v2 ==== 289+0+0 (1397559195 0 0) 0x2efab80 con 0x200d420 -6831> 2013-04-03 10:46:21.396489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21849 v187) v2 from mds.? 10.252.24.88:6800/17213 -6830> 2013-04-03 10:46:21.396503 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21849 v187) v2 -- ?+0 0x2a84600 con 0x200d420 -6829> 2013-04-03 10:46:21.635916 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114060 ==== forward(pg_stats(1 pgs tid 60 v 605) v1) to leader v1 ==== 753+0+0 (2263528561 0 0) 0x273bb80 con 0x208a840 -6828> 2013-04-03 10:46:21.635938 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -6827> 2013-04-03 10:46:21.635947 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -6826> 2013-04-03 10:46:21.636000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 preprocess_query pg_stats(1 pgs tid 60 v 605) v1 from osd.1 10.252.24.25:6805/3572 -6825> 2013-04-03 10:46:21.636025 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 prepare_update pg_stats(1 pgs tid 60 v 605) v1 from osd.1 10.252.24.25:6805/3572 -6824> 2013-04-03 10:46:21.636031 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 prepare_pg_stats pg_stats(1 pgs tid 60 v 605) v1 from osd.1 -6823> 2013-04-03 10:46:21.636037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29124 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -6822> 2013-04-03 10:46:21.686238 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29124 encode_pending v 29125 -6821> 2013-04-03 10:46:21.686283 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427527) queue_proposal bl 650 bytes; ctx = 0x2c3dcb0 -6820> 2013-04-03 10:46:21.686293 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427527) propose_queued 427528 650 bytes -6819> 2013-04-03 10:46:21.687444 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427527 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3222780 -6818> 2013-04-03 10:46:21.720223 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114061 ==== paxos(accept lc 427527 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1670558691 0 0) 0x3222780 con 0x208a840 -6817> 2013-04-03 10:46:21.721082 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427528 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3222c80 -6816> 2013-04-03 10:46:21.721103 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427528 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3222a00 -6815> 2013-04-03 10:46:21.721189 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29124 update_from_paxos applying incremental 29125 -6814> 2013-04-03 10:46:21.721211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 v29125: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1056B/s wr, 0op/s -6813> 2013-04-03 10:46:21.721229 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 send_pg_creates to 0 pgs -6812> 2013-04-03 10:46:21.721234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 update_logger -6811> 2013-04-03 10:46:21.721245 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 create_pending v 29126 -6810> 2013-04-03 10:46:21.721257 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29125 _updated_stats for osd.1 10.252.24.25:6805/3572 -6809> 2013-04-03 10:46:21.721264 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 60) v1 tid 17754) v2 -- ?+0 0x24f8000 con 0x208a840 -6808> 2013-04-03 10:46:21.721275 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 check_osd_map already seen 605 >= 605 -6807> 2013-04-03 10:46:21.721277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 update_logger -6806> 2013-04-03 10:46:21.721287 7fcdffcdb700 0 log [INF] : pgmap v29125: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1056B/s wr, 0op/s -6805> 2013-04-03 10:46:21.721288 7fcdffcdb700 10 send_log to self -6804> 2013-04-03 10:46:21.721289 7fcdffcdb700 10 log_queue is 1 last_log 191 sent 190 num 1 unsent 1 sending 1 -6803> 2013-04-03 10:46:21.721291 7fcdffcdb700 10 will send 2013-04-03 10:46:21.721288 mon.0 10.252.24.25:6789/0 191 : [INF] pgmap v29125: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1056B/s wr, 0op/s -6802> 2013-04-03 10:46:21.721300 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b76c00 -6801> 2013-04-03 10:46:21.721360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos -6800> 2013-04-03 10:46:21.721380 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos version 394115 summary v 394115 -6799> 2013-04-03 10:46:21.721398 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6798> 2013-04-03 10:46:21.721421 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76c00 con 0x1bac160 -6797> 2013-04-03 10:46:21.721444 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos -6796> 2013-04-03 10:46:21.721459 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos version 394115 summary v 394115 -6795> 2013-04-03 10:46:21.721470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6794> 2013-04-03 10:46:21.721480 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 preprocess_log log(1 entries) v1 from mon.0 -6793> 2013-04-03 10:46:21.721505 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6792> 2013-04-03 10:46:21.721517 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 prepare_log log(1 entries) v1 from mon.0 -6791> 2013-04-03 10:46:21.721527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 logging 2013-04-03 10:46:21.721288 mon.0 10.252.24.25:6789/0 191 : [INF] pgmap v29125: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1056B/s wr, 0op/s -6790> 2013-04-03 10:46:21.745636 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114062 ==== paxos(lease_ack lc 427528 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (807849650 0 0) 0x3222a00 con 0x208a840 -6789> 2013-04-03 10:46:21.745719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos -6788> 2013-04-03 10:46:21.745739 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394115 update_from_paxos version 394115 summary v 394115 -6787> 2013-04-03 10:46:21.745756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6786> 2013-04-03 10:46:22.720425 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394115 encode_pending v394116 -6785> 2013-04-03 10:46:22.720454 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427528) queue_proposal bl 443 bytes; ctx = 0x2c3dcd0 -6784> 2013-04-03 10:46:22.720464 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427528) propose_queued 427529 443 bytes -6783> 2013-04-03 10:46:22.721868 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427528 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3222500 -6782> 2013-04-03 10:46:22.769933 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114063 ==== paxos(accept lc 427528 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2274082170 0 0) 0x3222500 con 0x208a840 -6781> 2013-04-03 10:46:22.778840 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427529 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3222a00 -6780> 2013-04-03 10:46:22.778862 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427529 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3222780 -6779> 2013-04-03 10:46:22.778902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos -6778> 2013-04-03 10:46:22.778939 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394115 -6777> 2013-04-03 10:46:22.778954 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos latest full 393798 -6776> 2013-04-03 10:46:22.778987 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394116 update_from_paxos applying incremental log 394116 2013-04-03 10:46:21.721288 mon.0 10.252.24.25:6789/0 191 : [INF] pgmap v29125: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1056B/s wr, 0op/s -6775> 2013-04-03 10:46:22.779036 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 check_subs -6774> 2013-04-03 10:46:22.779069 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 create_pending v 394117 -6773> 2013-04-03 10:46:22.779092 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394116 _updated_log for mon.0 10.252.24.25:6789/0 -6772> 2013-04-03 10:46:22.779106 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 191) v1 -- ?+0 0x2e8b380 con 0x1bac160 -6771> 2013-04-03 10:46:22.779159 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos -6770> 2013-04-03 10:46:22.779174 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394116 -6769> 2013-04-03 10:46:22.779193 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6768> 2013-04-03 10:46:22.779214 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 191) v1 ==== 0+0+0 (0 0 0) 0x2e8b380 con 0x1bac160 -6767> 2013-04-03 10:46:22.779222 7fcdffcdb700 10 handle_log_ack log(last 191) v1 -6766> 2013-04-03 10:46:22.779225 7fcdffcdb700 10 logged 2013-04-03 10:46:21.721288 mon.0 10.252.24.25:6789/0 191 : [INF] pgmap v29125: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1056B/s wr, 0op/s -6765> 2013-04-03 10:46:22.795262 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114064 ==== paxos(lease_ack lc 427529 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (2311155441 0 0) 0x3222780 con 0x208a840 -6764> 2013-04-03 10:46:22.795306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos -6763> 2013-04-03 10:46:22.795323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394116 -6762> 2013-04-03 10:46:22.795338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6761> 2013-04-03 10:46:23.378816 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29125 check_down_pgs -6760> 2013-04-03 10:46:23.378850 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29125 v29125: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1056B/s wr, 0op/s -6759> 2013-04-03 10:46:23.378885 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -6758> 2013-04-03 10:46:23.378913 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -6757> 2013-04-03 10:46:23.378950 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -6756> 2013-04-03 10:46:23.378981 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos -6755> 2013-04-03 10:46:23.378999 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394116 -6754> 2013-04-03 10:46:23.379008 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394116 log -6753> 2013-04-03 10:46:23.379025 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6752> 2013-04-03 10:46:23.379044 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -6751> 2013-04-03 10:46:23.598621 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 195 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140479 v187) v2 ==== 289+0+0 (153905191 0 0) 0x273a000 con 0x2118420 -6750> 2013-04-03 10:46:23.598709 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140479 v187) v2 from mds.? 10.252.58.31:6800/10908 -6749> 2013-04-03 10:46:23.598724 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140479 v187) v2 -- ?+0 0x273bb80 con 0x2118420 -6748> 2013-04-03 10:46:24.945387 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114065 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 194 v187) v2) to leader v1 ==== 560+0+0 (879234691 0 0) 0x2a848c0 con 0x208a840 -6747> 2013-04-03 10:46:24.945409 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -6746> 2013-04-03 10:46:24.945419 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4dc0 from 10.252.24.88:6789/0 -6745> 2013-04-03 10:46:24.945478 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 194 v187) v2 from mds.0 10.252.24.25:6802/3144 -6744> 2013-04-03 10:46:24.945491 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 194 v187) v2 tid 17755) v2 -- ?+0 0x24f8d80 con 0x208a840 -6743> 2013-04-03 10:46:25.396465 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 196 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21850 v187) v2 ==== 289+0+0 (922668572 0 0) 0x2ef9dc0 con 0x200d420 -6742> 2013-04-03 10:46:25.396547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21850 v187) v2 from mds.? 10.252.24.88:6800/17213 -6741> 2013-04-03 10:46:25.396560 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21850 v187) v2 -- ?+0 0x2a848c0 con 0x200d420 -6740> 2013-04-03 10:46:25.486845 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 196 ==== mon_subscribe({mdsmap=188+,monmap=6+}) v2 ==== 42+0+0 (435898880 0 0) 0x2e8b540 con 0x2118420 -6739> 2013-04-03 10:46:25.486874 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 -6738> 2013-04-03 10:46:25.486886 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -6737> 2013-04-03 10:46:25.486888 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.58.31:6800/10908 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8b380 -6736> 2013-04-03 10:46:25.778931 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427529 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3222280 -6735> 2013-04-03 10:46:25.779686 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114066 ==== paxos(lease_ack lc 427529 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (1572932308 0 0) 0x3222280 con 0x208a840 -6734> 2013-04-03 10:46:25.779766 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos -6733> 2013-04-03 10:46:25.779786 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394116 -6732> 2013-04-03 10:46:25.779804 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6731> 2013-04-03 10:46:26.636477 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114067 ==== forward(pg_stats(1 pgs tid 61 v 605) v1) to leader v1 ==== 753+0+0 (2336221325 0 0) 0x273a000 con 0x208a840 -6730> 2013-04-03 10:46:26.636497 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -6729> 2013-04-03 10:46:26.636507 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -6728> 2013-04-03 10:46:26.636561 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 preprocess_query pg_stats(1 pgs tid 61 v 605) v1 from osd.1 10.252.24.25:6805/3572 -6727> 2013-04-03 10:46:26.636586 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 prepare_update pg_stats(1 pgs tid 61 v 605) v1 from osd.1 10.252.24.25:6805/3572 -6726> 2013-04-03 10:46:26.636592 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 prepare_pg_stats pg_stats(1 pgs tid 61 v 605) v1 from osd.1 -6725> 2013-04-03 10:46:26.636598 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29125 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -6724> 2013-04-03 10:46:26.686780 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29125 encode_pending v 29126 -6723> 2013-04-03 10:46:26.686824 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427529) queue_proposal bl 650 bytes; ctx = 0x2c3dbb0 -6722> 2013-04-03 10:46:26.686833 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427529) propose_queued 427530 650 bytes -6721> 2013-04-03 10:46:26.688428 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427529 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3222000 -6720> 2013-04-03 10:46:26.734157 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114068 ==== paxos(accept lc 427529 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1071907373 0 0) 0x3222000 con 0x208a840 -6719> 2013-04-03 10:46:26.734511 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427530 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3222280 -6718> 2013-04-03 10:46:26.734531 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427530 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3222780 -6717> 2013-04-03 10:46:26.734619 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29125 update_from_paxos applying incremental 29126 -6716> 2013-04-03 10:46:26.734640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 v29126: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6715> 2013-04-03 10:46:26.734659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 send_pg_creates to 0 pgs -6714> 2013-04-03 10:46:26.734663 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 update_logger -6713> 2013-04-03 10:46:26.734674 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 create_pending v 29127 -6712> 2013-04-03 10:46:26.734683 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29126 _updated_stats for osd.1 10.252.24.25:6805/3572 -6711> 2013-04-03 10:46:26.734690 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 61) v1 tid 17756) v2 -- ?+0 0x24f8000 con 0x208a840 -6710> 2013-04-03 10:46:26.734700 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 check_osd_map already seen 605 >= 605 -6709> 2013-04-03 10:46:26.734703 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 update_logger -6708> 2013-04-03 10:46:26.734714 7fcdffcdb700 0 log [INF] : pgmap v29126: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6707> 2013-04-03 10:46:26.734716 7fcdffcdb700 10 send_log to self -6706> 2013-04-03 10:46:26.734716 7fcdffcdb700 10 log_queue is 1 last_log 192 sent 191 num 1 unsent 1 sending 1 -6705> 2013-04-03 10:46:26.734718 7fcdffcdb700 10 will send 2013-04-03 10:46:26.734715 mon.0 10.252.24.25:6789/0 192 : [INF] pgmap v29126: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6704> 2013-04-03 10:46:26.734726 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b76c00 -6703> 2013-04-03 10:46:26.734776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos -6702> 2013-04-03 10:46:26.734795 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394116 -6701> 2013-04-03 10:46:26.734814 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6700> 2013-04-03 10:46:26.734838 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76c00 con 0x1bac160 -6699> 2013-04-03 10:46:26.734860 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos -6698> 2013-04-03 10:46:26.734874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394116 -6697> 2013-04-03 10:46:26.734883 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6696> 2013-04-03 10:46:26.734896 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 preprocess_log log(1 entries) v1 from mon.0 -6695> 2013-04-03 10:46:26.734919 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6694> 2013-04-03 10:46:26.734931 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 prepare_log log(1 entries) v1 from mon.0 -6693> 2013-04-03 10:46:26.734941 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 logging 2013-04-03 10:46:26.734715 mon.0 10.252.24.25:6789/0 192 : [INF] pgmap v29126: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6692> 2013-04-03 10:46:26.745892 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114069 ==== paxos(lease_ack lc 427530 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3823312832 0 0) 0x3222780 con 0x208a840 -6691> 2013-04-03 10:46:26.745941 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos -6690> 2013-04-03 10:46:26.745959 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394116 update_from_paxos version 394116 summary v 394116 -6689> 2013-04-03 10:46:26.745974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6688> 2013-04-03 10:46:27.598739 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 197 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140480 v187) v2 ==== 289+0+0 (1277534163 0 0) 0x273b8c0 con 0x2118420 -6687> 2013-04-03 10:46:27.598822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140480 v187) v2 from mds.? 10.252.58.31:6800/10908 -6686> 2013-04-03 10:46:27.598836 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140480 v187) v2 -- ?+0 0x273a000 con 0x2118420 -6685> 2013-04-03 10:46:27.734356 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394116 encode_pending v394117 -6684> 2013-04-03 10:46:27.734384 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427530) queue_proposal bl 442 bytes; ctx = 0x2c3dcf0 -6683> 2013-04-03 10:46:27.734673 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427530) propose_queued 427531 442 bytes -6682> 2013-04-03 10:46:27.735709 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427530 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4c80 -6681> 2013-04-03 10:46:27.761302 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114070 ==== paxos(accept lc 427530 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (249117018 0 0) 0x2fe4c80 con 0x208a840 -6680> 2013-04-03 10:46:27.761665 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427531 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3222780 -6679> 2013-04-03 10:46:27.761687 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427531 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3222000 -6678> 2013-04-03 10:46:27.761725 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos -6677> 2013-04-03 10:46:27.761762 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos version 394117 summary v 394116 -6676> 2013-04-03 10:46:27.761776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos latest full 393798 -6675> 2013-04-03 10:46:27.761794 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394117 update_from_paxos applying incremental log 394117 2013-04-03 10:46:26.734715 mon.0 10.252.24.25:6789/0 192 : [INF] pgmap v29126: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6674> 2013-04-03 10:46:27.761854 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 check_subs -6673> 2013-04-03 10:46:27.761888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 create_pending v 394118 -6672> 2013-04-03 10:46:27.761910 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394117 _updated_log for mon.0 10.252.24.25:6789/0 -6671> 2013-04-03 10:46:27.761923 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 192) v1 -- ?+0 0x2e8b8c0 con 0x1bac160 -6670> 2013-04-03 10:46:27.761976 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos -6669> 2013-04-03 10:46:27.761992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos version 394117 summary v 394117 -6668> 2013-04-03 10:46:27.762010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6667> 2013-04-03 10:46:27.762032 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 192) v1 ==== 0+0+0 (0 0 0) 0x2e8b8c0 con 0x1bac160 -6666> 2013-04-03 10:46:27.762039 7fcdffcdb700 10 handle_log_ack log(last 192) v1 -6665> 2013-04-03 10:46:27.762042 7fcdffcdb700 10 logged 2013-04-03 10:46:26.734715 mon.0 10.252.24.25:6789/0 192 : [INF] pgmap v29126: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6664> 2013-04-03 10:46:27.773060 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114071 ==== paxos(lease_ack lc 427531 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (733859275 0 0) 0x3222000 con 0x208a840 -6663> 2013-04-03 10:46:27.773098 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos -6662> 2013-04-03 10:46:27.773115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos version 394117 summary v 394117 -6661> 2013-04-03 10:46:27.773524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6660> 2013-04-03 10:46:28.283223 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114072 ==== forward(pg_stats(1 pgs tid 43 v 605) v1) to leader v1 ==== 757+0+0 (3186114273 0 0) 0x2cc5b80 con 0x208a840 -6659> 2013-04-03 10:46:28.283252 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -6658> 2013-04-03 10:46:28.283265 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 -6657> 2013-04-03 10:46:28.283318 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 preprocess_query pg_stats(1 pgs tid 43 v 605) v1 from osd.2 10.252.24.30:6801/5656 -6656> 2013-04-03 10:46:28.283343 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 prepare_update pg_stats(1 pgs tid 43 v 605) v1 from osd.2 10.252.24.30:6801/5656 -6655> 2013-04-03 10:46:28.283348 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 prepare_pg_stats pg_stats(1 pgs tid 43 v 605) v1 from osd.2 -6654> 2013-04-03 10:46:28.283355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29126 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -6653> 2013-04-03 10:46:28.379176 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29126 check_down_pgs -6652> 2013-04-03 10:46:28.379207 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29126 v29126: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6651> 2013-04-03 10:46:28.379239 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -6650> 2013-04-03 10:46:28.379264 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -6649> 2013-04-03 10:46:28.379301 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -6648> 2013-04-03 10:46:28.379703 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos -6647> 2013-04-03 10:46:28.379725 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos version 394117 summary v 394117 -6646> 2013-04-03 10:46:28.379734 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394117 log -6645> 2013-04-03 10:46:28.379751 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6644> 2013-04-03 10:46:28.379772 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -6643> 2013-04-03 10:46:28.761500 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29126 encode_pending v 29127 -6642> 2013-04-03 10:46:28.761547 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427531) queue_proposal bl 654 bytes; ctx = 0x2c3dca0 -6641> 2013-04-03 10:46:28.761555 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427469..427531) propose_queued 427532 654 bytes -6640> 2013-04-03 10:46:28.763893 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427531 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3250c80 -6639> 2013-04-03 10:46:28.811260 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114073 ==== paxos(accept lc 427531 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4291337239 0 0) 0x3250c80 con 0x208a840 -6638> 2013-04-03 10:46:28.811572 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427532 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3222000 -6637> 2013-04-03 10:46:28.811593 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427532 fc 427469 pn 0 opn 0) v3 -- ?+0 0x2fe4c80 -6636> 2013-04-03 10:46:28.811683 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29126 update_from_paxos applying incremental 29127 -6635> 2013-04-03 10:46:28.811706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29127 v29127: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1736B/s wr, 0op/s -6634> 2013-04-03 10:46:28.811726 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29127 send_pg_creates to 0 pgs -6633> 2013-04-03 10:46:28.811729 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29127 update_logger -6632> 2013-04-03 10:46:28.811739 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29127 create_pending v 29128 -6631> 2013-04-03 10:46:28.811748 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29127 _updated_stats for osd.2 10.252.24.30:6801/5656 -6630> 2013-04-03 10:46:28.811754 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 43) v1 tid 17757) v2 -- ?+0 0x24f8d80 con 0x208a840 -6629> 2013-04-03 10:46:28.811765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29127 check_osd_map already seen 605 >= 605 -6628> 2013-04-03 10:46:28.811767 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29127 update_logger -6627> 2013-04-03 10:46:28.811777 7fcdffcdb700 0 log [INF] : pgmap v29127: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1736B/s wr, 0op/s -6626> 2013-04-03 10:46:28.811779 7fcdffcdb700 10 send_log to self -6625> 2013-04-03 10:46:28.811787 7fcdffcdb700 10 log_queue is 1 last_log 193 sent 192 num 1 unsent 1 sending 1 -6624> 2013-04-03 10:46:28.811788 7fcdffcdb700 10 will send 2013-04-03 10:46:28.811778 mon.0 10.252.24.25:6789/0 193 : [INF] pgmap v29127: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1736B/s wr, 0op/s -6623> 2013-04-03 10:46:28.811797 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b76c00 -6622> 2013-04-03 10:46:28.811850 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos -6621> 2013-04-03 10:46:28.811869 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos version 394117 summary v 394117 -6620> 2013-04-03 10:46:28.811889 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6619> 2013-04-03 10:46:28.811911 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76c00 con 0x1bac160 -6618> 2013-04-03 10:46:28.811934 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos -6617> 2013-04-03 10:46:28.811948 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos version 394117 summary v 394117 -6616> 2013-04-03 10:46:28.811957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6615> 2013-04-03 10:46:28.811969 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 preprocess_log log(1 entries) v1 from mon.0 -6614> 2013-04-03 10:46:28.811994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6613> 2013-04-03 10:46:28.812006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 prepare_log log(1 entries) v1 from mon.0 -6612> 2013-04-03 10:46:28.812018 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 logging 2013-04-03 10:46:28.811778 mon.0 10.252.24.25:6789/0 193 : [INF] pgmap v29127: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1736B/s wr, 0op/s -6611> 2013-04-03 10:46:28.824244 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114074 ==== paxos(lease_ack lc 427532 fc 427469 pn 0 opn 0) v3 ==== 80+0+0 (3693120474 0 0) 0x2fe4c80 con 0x208a840 -6610> 2013-04-03 10:46:28.824306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos -6609> 2013-04-03 10:46:28.824323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394117 update_from_paxos version 394117 summary v 394117 -6608> 2013-04-03 10:46:28.824339 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -6607> 2013-04-03 10:46:28.871664 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427469..427532) queue_proposal bl 1647 bytes; ctx = 0x2c3d970 -6606> 2013-04-03 10:46:28.927584 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114075 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (424405211 0 0) 0x2cc5340 con 0x208a840 -6605> 2013-04-03 10:46:28.927618 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -6604> 2013-04-03 10:46:28.927625 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -6603> 2013-04-03 10:46:28.927634 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 -6602> 2013-04-03 10:46:28.927639 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( start_reply flags( reply-to(mon.2 10.252.24.88:6789/0) ) ) v1 -- ?+0 0x2cc5b80 -6601> 2013-04-03 10:46:28.945419 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114076 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 195 v187) v2) to leader v1 ==== 560+0+0 (144474809 0 0) 0x2cc4840 con 0x208a840 -6600> 2013-04-03 10:46:28.945441 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -6599> 2013-04-03 10:46:28.945451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc4b00 from 10.252.24.88:6789/0 -6598> 2013-04-03 10:46:28.945506 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 195 v187) v2 from mds.0 10.252.24.25:6802/3144 -6597> 2013-04-03 10:46:28.945519 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 195 v187) v2 tid 17758) v2 -- ?+0 0x24f8000 con 0x208a840 -6596> 2013-04-03 10:46:29.396615 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 197 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21851 v187) v2 ==== 289+0+0 (361787745 0 0) 0x2ef92c0 con 0x200d420 -6595> 2013-04-03 10:46:29.396698 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21851 v187) v2 from mds.? 10.252.24.88:6800/17213 -6594> 2013-04-03 10:46:29.396714 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21851 v187) v2 -- ?+0 0x2cc4840 con 0x200d420 -6593> 2013-04-03 10:46:29.811472 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394117 encode_pending v394118 -6592> 2013-04-03 10:46:29.811503 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427469..427532) queue_proposal bl 443 bytes; ctx = 0x2c3dd10 -6591> 2013-04-03 10:46:29.811511 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427469..427532) propose_queued 427533 1647 bytes -6590> 2013-04-03 10:46:29.812574 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427532 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3250a00 -6589> 2013-04-03 10:46:29.848135 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114077 ==== paxos(accept lc 427532 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2794018271 0 0) 0x3250a00 con 0x208a840 -6588> 2013-04-03 10:46:29.848598 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427533 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4c80 -6587> 2013-04-03 10:46:29.848625 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427533 fc 427469 pn 0 opn 0) v3 -- ?+0 0x3250c80 -6586> 2013-04-03 10:46:29.848701 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427533) propose_queued 427534 443 bytes -6585> 2013-04-03 10:46:29.849504 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427533 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3222500 -6584> 2013-04-03 10:46:29.859799 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114078 ==== paxos(lease_ack lc 427533 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (4268542508 0 0) 0x3222500 con 0x208a840 -6583> 2013-04-03 10:46:29.871496 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114079 ==== paxos(accept lc 427533 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1947316878 0 0) 0x3250c80 con 0x208a840 -6582> 2013-04-03 10:46:29.871822 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427534 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3222500 -6581> 2013-04-03 10:46:29.871846 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427534 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250a00 -6580> 2013-04-03 10:46:29.871888 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos -6579> 2013-04-03 10:46:29.871933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394117 -6578> 2013-04-03 10:46:29.871948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos latest full 393798 -6577> 2013-04-03 10:46:29.872624 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos applying incremental log 394118 2013-04-03 10:46:28.811778 mon.0 10.252.24.25:6789/0 193 : [INF] pgmap v29127: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1736B/s wr, 0op/s -6576> 2013-04-03 10:46:29.872670 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 check_subs -6575> 2013-04-03 10:46:29.872713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 create_pending v 394119 -6574> 2013-04-03 10:46:29.872739 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 _updated_log for mon.0 10.252.24.25:6789/0 -6573> 2013-04-03 10:46:29.872753 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 193) v1 -- ?+0 0x2e8ba80 con 0x1bac160 -6572> 2013-04-03 10:46:29.872810 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos -6571> 2013-04-03 10:46:29.872827 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 -6570> 2013-04-03 10:46:29.872846 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6569> 2013-04-03 10:46:29.872869 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 193) v1 ==== 0+0+0 (0 0 0) 0x2e8ba80 con 0x1bac160 -6568> 2013-04-03 10:46:29.872877 7fcdffcdb700 10 handle_log_ack log(last 193) v1 -6567> 2013-04-03 10:46:29.872880 7fcdffcdb700 10 logged 2013-04-03 10:46:28.811778 mon.0 10.252.24.25:6789/0 193 : [INF] pgmap v29127: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1736B/s wr, 0op/s -6566> 2013-04-03 10:46:29.896473 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114080 ==== paxos(lease_ack lc 427534 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3119514769 0 0) 0x3250a00 con 0x208a840 -6565> 2013-04-03 10:46:29.896563 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos -6564> 2013-04-03 10:46:29.896584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 -6563> 2013-04-03 10:46:29.896659 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6562> 2013-04-03 10:46:31.257210 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2fe4000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33175/0 -6561> 2013-04-03 10:46:31.257248 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2fe4000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33175/0) -6560> 2013-04-03 10:46:31.257263 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2fe4000 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -6559> 2013-04-03 10:46:31.257456 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x2fe4000 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -6558> 2013-04-03 10:46:31.598908 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 198 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140481 v187) v2 ==== 289+0+0 (1867194542 0 0) 0x2ce1000 con 0x2118420 -6557> 2013-04-03 10:46:31.599010 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140481 v187) v2 from mds.? 10.252.58.31:6800/10908 -6556> 2013-04-03 10:46:31.599028 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140481 v187) v2 -- ?+0 0x2ef92c0 con 0x2118420 -6555> 2013-04-03 10:46:32.871922 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427534 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250780 -6554> 2013-04-03 10:46:32.872710 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114081 ==== paxos(lease_ack lc 427534 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (2722748128 0 0) 0x3250780 con 0x208a840 -6553> 2013-04-03 10:46:32.872786 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos -6552> 2013-04-03 10:46:32.872807 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 -6551> 2013-04-03 10:46:32.872826 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6550> 2013-04-03 10:46:32.945597 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114082 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 196 v187) v2) to leader v1 ==== 560+0+0 (2616482029 0 0) 0x2cc5340 con 0x208a840 -6549> 2013-04-03 10:46:32.945619 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -6548> 2013-04-03 10:46:32.945631 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc5080 from 10.252.24.88:6789/0 -6547> 2013-04-03 10:46:32.945684 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 196 v187) v2 from mds.0 10.252.24.25:6802/3144 -6546> 2013-04-03 10:46:32.945697 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 196 v187) v2 tid 17759) v2 -- ?+0 0x24f8d80 con 0x208a840 -6545> 2013-04-03 10:46:33.379900 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 check_down_pgs -6544> 2013-04-03 10:46:33.379934 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 v29127: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1736B/s wr, 0op/s -6543> 2013-04-03 10:46:33.379964 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -6542> 2013-04-03 10:46:33.379990 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -6541> 2013-04-03 10:46:33.380028 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -6540> 2013-04-03 10:46:33.380060 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos -6539> 2013-04-03 10:46:33.380080 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 -6538> 2013-04-03 10:46:33.380090 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 log -6537> 2013-04-03 10:46:33.380107 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6536> 2013-04-03 10:46:33.380125 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -6535> 2013-04-03 10:46:33.396735 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 198 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21852 v187) v2 ==== 289+0+0 (4254682386 0 0) 0x2efa8c0 con 0x200d420 -6534> 2013-04-03 10:46:33.396814 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21852 v187) v2 from mds.? 10.252.24.88:6800/17213 -6533> 2013-04-03 10:46:33.396829 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21852 v187) v2 -- ?+0 0x2cc5340 con 0x200d420 -6532> 2013-04-03 10:46:35.598993 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 199 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140482 v187) v2 ==== 289+0+0 (184082729 0 0) 0x2ce1840 con 0x2118420 -6531> 2013-04-03 10:46:35.599076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140482 v187) v2 from mds.? 10.252.58.31:6800/10908 -6530> 2013-04-03 10:46:35.599090 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140482 v187) v2 -- ?+0 0x2efa8c0 con 0x2118420 -6529> 2013-04-03 10:46:35.871998 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427534 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250500 -6528> 2013-04-03 10:46:35.872767 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114083 ==== paxos(lease_ack lc 427534 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1563394716 0 0) 0x3250500 con 0x208a840 -6527> 2013-04-03 10:46:35.872842 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos -6526> 2013-04-03 10:46:35.872863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 -6525> 2013-04-03 10:46:35.872881 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6524> 2013-04-03 10:46:36.637971 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114084 ==== forward(pg_stats(1 pgs tid 62 v 605) v1) to leader v1 ==== 753+0+0 (1174839856 0 0) 0x2ce1000 con 0x208a840 -6523> 2013-04-03 10:46:36.637992 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -6522> 2013-04-03 10:46:36.638004 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -6521> 2013-04-03 10:46:36.638061 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 preprocess_query pg_stats(1 pgs tid 62 v 605) v1 from osd.1 10.252.24.25:6805/3572 -6520> 2013-04-03 10:46:36.638086 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 prepare_update pg_stats(1 pgs tid 62 v 605) v1 from osd.1 10.252.24.25:6805/3572 -6519> 2013-04-03 10:46:36.638093 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 prepare_pg_stats pg_stats(1 pgs tid 62 v 605) v1 from osd.1 -6518> 2013-04-03 10:46:36.638100 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -6517> 2013-04-03 10:46:36.688283 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 encode_pending v 29128 -6516> 2013-04-03 10:46:36.688332 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427534) queue_proposal bl 650 bytes; ctx = 0x2c3db90 -6515> 2013-04-03 10:46:36.688342 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427534) propose_queued 427535 650 bytes -6514> 2013-04-03 10:46:36.689270 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427534 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3250280 -6513> 2013-04-03 10:46:36.714824 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114085 ==== paxos(accept lc 427534 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (971606437 0 0) 0x3250280 con 0x208a840 -6512> 2013-04-03 10:46:36.725850 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427535 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3250500 -6511> 2013-04-03 10:46:36.725871 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427535 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250780 -6510> 2013-04-03 10:46:36.725961 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29127 update_from_paxos applying incremental 29128 -6509> 2013-04-03 10:46:36.725983 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 v29128: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -6508> 2013-04-03 10:46:36.726003 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 send_pg_creates to 0 pgs -6507> 2013-04-03 10:46:36.726006 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 update_logger -6506> 2013-04-03 10:46:36.726017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 create_pending v 29129 -6505> 2013-04-03 10:46:36.726033 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 _updated_stats for osd.1 10.252.24.25:6805/3572 -6504> 2013-04-03 10:46:36.726041 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 62) v1 tid 17760) v2 -- ?+0 0x24f8000 con 0x208a840 -6503> 2013-04-03 10:46:36.726051 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 check_osd_map already seen 605 >= 605 -6502> 2013-04-03 10:46:36.726054 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 update_logger -6501> 2013-04-03 10:46:36.726065 7fcdffcdb700 0 log [INF] : pgmap v29128: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -6500> 2013-04-03 10:46:36.726066 7fcdffcdb700 10 send_log to self -6499> 2013-04-03 10:46:36.726067 7fcdffcdb700 10 log_queue is 1 last_log 194 sent 193 num 1 unsent 1 sending 1 -6498> 2013-04-03 10:46:36.726069 7fcdffcdb700 10 will send 2013-04-03 10:46:36.726066 mon.0 10.252.24.25:6789/0 194 : [INF] pgmap v29128: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -6497> 2013-04-03 10:46:36.726076 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b76c00 -6496> 2013-04-03 10:46:36.726128 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos -6495> 2013-04-03 10:46:36.726148 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 -6494> 2013-04-03 10:46:36.726167 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6493> 2013-04-03 10:46:36.726187 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76c00 con 0x1bac160 -6492> 2013-04-03 10:46:36.726209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos -6491> 2013-04-03 10:46:36.726224 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 -6490> 2013-04-03 10:46:36.726233 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6489> 2013-04-03 10:46:36.726248 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 preprocess_log log(1 entries) v1 from mon.0 -6488> 2013-04-03 10:46:36.726275 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6487> 2013-04-03 10:46:36.726288 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 prepare_log log(1 entries) v1 from mon.0 -6486> 2013-04-03 10:46:36.726301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 logging 2013-04-03 10:46:36.726066 mon.0 10.252.24.25:6789/0 194 : [INF] pgmap v29128: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -6485> 2013-04-03 10:46:36.734822 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114086 ==== paxos(lease_ack lc 427535 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3387438112 0 0) 0x3250780 con 0x208a840 -6484> 2013-04-03 10:46:36.734873 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos -6483> 2013-04-03 10:46:36.734893 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 update_from_paxos version 394118 summary v 394118 -6482> 2013-04-03 10:46:36.734909 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6481> 2013-04-03 10:46:36.803420 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114087 ==== forward(pg_stats(1 pgs tid 30462 v 605) v1) to leader v1 ==== 757+0+0 (2587630235 0 0) 0x2cc58c0 con 0x208a840 -6480> 2013-04-03 10:46:36.803449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -6479> 2013-04-03 10:46:36.803463 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 -6478> 2013-04-03 10:46:36.803516 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 preprocess_query pg_stats(1 pgs tid 30462 v 605) v1 from osd.3 10.252.24.88:6801/17538 -6477> 2013-04-03 10:46:36.803542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 prepare_update pg_stats(1 pgs tid 30462 v 605) v1 from osd.3 10.252.24.88:6801/17538 -6476> 2013-04-03 10:46:36.803548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 prepare_pg_stats pg_stats(1 pgs tid 30462 v 605) v1 from osd.3 -6475> 2013-04-03 10:46:36.803555 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -6474> 2013-04-03 10:46:36.945779 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114088 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 197 v187) v2) to leader v1 ==== 560+0+0 (2995380088 0 0) 0x2a84340 con 0x208a840 -6473> 2013-04-03 10:46:36.945801 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -6472> 2013-04-03 10:46:36.945812 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a83580 from 10.252.24.88:6789/0 -6471> 2013-04-03 10:46:36.945865 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 197 v187) v2 from mds.0 10.252.24.25:6802/3144 -6470> 2013-04-03 10:46:36.945878 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 197 v187) v2 tid 17762) v2 -- ?+0 0x24f8d80 con 0x208a840 -6469> 2013-04-03 10:46:37.396830 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 199 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21853 v187) v2 ==== 289+0+0 (3740791407 0 0) 0x2ef9840 con 0x200d420 -6468> 2013-04-03 10:46:37.396914 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21853 v187) v2 from mds.? 10.252.24.88:6800/17213 -6467> 2013-04-03 10:46:37.396934 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21853 v187) v2 -- ?+0 0x2a84340 con 0x200d420 -6466> 2013-04-03 10:46:37.715024 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 encode_pending v 29129 -6465> 2013-04-03 10:46:37.715078 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427535) queue_proposal bl 654 bytes; ctx = 0x2c3ded0 -6464> 2013-04-03 10:46:37.715089 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427535) propose_queued 427536 654 bytes -6463> 2013-04-03 10:46:37.716425 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427535 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3250000 -6462> 2013-04-03 10:46:37.716509 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394118 encode_pending v394119 -6461> 2013-04-03 10:46:37.716537 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427522..427535) queue_proposal bl 443 bytes; ctx = 0x2c3dee0 -6460> 2013-04-03 10:46:37.716544 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427522..427535) propose_new_value not active; proposal queued -6459> 2013-04-03 10:46:37.742076 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114089 ==== paxos(accept lc 427535 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (824929231 0 0) 0x3250000 con 0x208a840 -6458> 2013-04-03 10:46:37.742467 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427536 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3250780 -6457> 2013-04-03 10:46:37.742489 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427536 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250280 -6456> 2013-04-03 10:46:37.742576 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29128 update_from_paxos applying incremental 29129 -6455> 2013-04-03 10:46:37.742598 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 v29129: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 571B/s wr, 0op/s -6454> 2013-04-03 10:46:37.742617 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 send_pg_creates to 0 pgs -6453> 2013-04-03 10:46:37.742620 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 update_logger -6452> 2013-04-03 10:46:37.742631 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 create_pending v 29130 -6451> 2013-04-03 10:46:37.742640 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 _updated_stats for osd.3 10.252.24.88:6801/17538 -6450> 2013-04-03 10:46:37.742646 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30462) v1 tid 17761) v2 -- ?+0 0x2b26b40 con 0x208a840 -6449> 2013-04-03 10:46:37.742657 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 check_osd_map already seen 605 >= 605 -6448> 2013-04-03 10:46:37.742661 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 update_logger -6447> 2013-04-03 10:46:37.742672 7fcdffcdb700 0 log [INF] : pgmap v29129: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 571B/s wr, 0op/s -6446> 2013-04-03 10:46:37.742673 7fcdffcdb700 10 send_log to self -6445> 2013-04-03 10:46:37.742674 7fcdffcdb700 10 log_queue is 2 last_log 195 sent 194 num 2 unsent 1 sending 1 -6444> 2013-04-03 10:46:37.742676 7fcdffcdb700 10 will send 2013-04-03 10:46:36.726066 mon.0 10.252.24.25:6789/0 194 : [INF] pgmap v29128: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -6443> 2013-04-03 10:46:37.742685 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75600 -6442> 2013-04-03 10:46:37.742711 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427536) propose_queued 427537 443 bytes -6441> 2013-04-03 10:46:37.742882 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427536 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3250a00 -6440> 2013-04-03 10:46:37.742899 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75600 con 0x1bac160 -6439> 2013-04-03 10:46:37.766847 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114090 ==== paxos(lease_ack lc 427536 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3067284424 0 0) 0x3250a00 con 0x208a840 -6438> 2013-04-03 10:46:37.778448 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114091 ==== paxos(accept lc 427536 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (680308758 0 0) 0x2116500 con 0x208a840 -6437> 2013-04-03 10:46:37.779692 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427537 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3250a00 -6436> 2013-04-03 10:46:37.779725 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427537 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250000 -6435> 2013-04-03 10:46:37.779768 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos -6434> 2013-04-03 10:46:37.779806 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394118 -6433> 2013-04-03 10:46:37.779820 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos latest full 393798 -6432> 2013-04-03 10:46:37.779842 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos applying incremental log 394119 2013-04-03 10:46:36.726066 mon.0 10.252.24.25:6789/0 194 : [INF] pgmap v29128: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -6431> 2013-04-03 10:46:37.779894 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 check_subs -6430> 2013-04-03 10:46:37.779926 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 create_pending v 394120 -6429> 2013-04-03 10:46:37.779949 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 _updated_log for mon.0 10.252.24.25:6789/0 -6428> 2013-04-03 10:46:37.779962 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 194) v1 -- ?+0 0x2e8be00 con 0x1bac160 -6427> 2013-04-03 10:46:37.779983 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos -6426> 2013-04-03 10:46:37.779998 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 -6425> 2013-04-03 10:46:37.780008 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6424> 2013-04-03 10:46:37.780021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 preprocess_log log(1 entries) v1 from mon.0 -6423> 2013-04-03 10:46:37.780040 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 nothing new -6422> 2013-04-03 10:46:37.780096 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos -6421> 2013-04-03 10:46:37.780112 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 -6420> 2013-04-03 10:46:37.780132 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6419> 2013-04-03 10:46:37.780154 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 194) v1 ==== 0+0+0 (0 0 0) 0x2e8be00 con 0x1bac160 -6418> 2013-04-03 10:46:37.780162 7fcdffcdb700 10 handle_log_ack log(last 194) v1 -6417> 2013-04-03 10:46:37.780165 7fcdffcdb700 10 logged 2013-04-03 10:46:36.726066 mon.0 10.252.24.25:6789/0 194 : [INF] pgmap v29128: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1228B/s wr, 0op/s -6416> 2013-04-03 10:46:37.791561 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114092 ==== paxos(lease_ack lc 427537 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (723574784 0 0) 0x3250000 con 0x208a840 -6415> 2013-04-03 10:46:37.791706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos -6414> 2013-04-03 10:46:37.791734 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 -6413> 2013-04-03 10:46:37.791752 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6412> 2013-04-03 10:46:38.380251 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 check_down_pgs -6411> 2013-04-03 10:46:38.380286 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 v29129: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 571B/s wr, 0op/s -6410> 2013-04-03 10:46:38.380316 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -6409> 2013-04-03 10:46:38.380341 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -6408> 2013-04-03 10:46:38.380376 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -6407> 2013-04-03 10:46:38.380409 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos -6406> 2013-04-03 10:46:38.380429 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 -6405> 2013-04-03 10:46:38.380439 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 log -6404> 2013-04-03 10:46:38.380456 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6403> 2013-04-03 10:46:38.380476 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -6402> 2013-04-03 10:46:39.599083 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 200 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140483 v187) v2 ==== 289+0+0 (697720404 0 0) 0x2ce2b80 con 0x2118420 -6401> 2013-04-03 10:46:39.599168 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140483 v187) v2 from mds.? 10.252.58.31:6800/10908 -6400> 2013-04-03 10:46:39.599183 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140483 v187) v2 -- ?+0 0x2ef9840 con 0x2118420 -6399> 2013-04-03 10:46:40.779812 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427537 fc 427522 pn 0 opn 0) v3 -- ?+0 0x2fe4000 -6398> 2013-04-03 10:46:40.780605 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114093 ==== paxos(lease_ack lc 427537 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (224063246 0 0) 0x2fe4000 con 0x208a840 -6397> 2013-04-03 10:46:40.780675 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos -6396> 2013-04-03 10:46:40.780696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 -6395> 2013-04-03 10:46:40.780714 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6394> 2013-04-03 10:46:40.945890 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114094 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 198 v187) v2) to leader v1 ==== 560+0+0 (830156569 0 0) 0x2cc58c0 con 0x208a840 -6393> 2013-04-03 10:46:40.945912 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -6392> 2013-04-03 10:46:40.945925 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273a580 from 10.252.24.88:6789/0 -6391> 2013-04-03 10:46:40.945979 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 198 v187) v2 from mds.0 10.252.24.25:6802/3144 -6390> 2013-04-03 10:46:40.945991 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 198 v187) v2 tid 17763) v2 -- ?+0 0x24f8000 con 0x208a840 -6389> 2013-04-03 10:46:41.273250 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x300c280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33176/0 -6388> 2013-04-03 10:46:41.273288 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x300c280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33176/0) -6387> 2013-04-03 10:46:41.273302 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x300c280 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -6386> 2013-04-03 10:46:41.273605 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x300c280 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -6385> 2013-04-03 10:46:41.396965 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 200 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21854 v187) v2 ==== 289+0+0 (3141809128 0 0) 0x2ef9580 con 0x200d420 -6384> 2013-04-03 10:46:41.397048 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21854 v187) v2 from mds.? 10.252.24.88:6800/17213 -6383> 2013-04-03 10:46:41.397063 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21854 v187) v2 -- ?+0 0x2cc58c0 con 0x200d420 -6382> 2013-04-03 10:46:43.285127 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114095 ==== forward(pg_stats(1 pgs tid 44 v 605) v1) to leader v1 ==== 757+0+0 (3137793411 0 0) 0x2ce2b80 con 0x208a840 -6381> 2013-04-03 10:46:43.285153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -6380> 2013-04-03 10:46:43.285165 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 -6379> 2013-04-03 10:46:43.285216 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 preprocess_query pg_stats(1 pgs tid 44 v 605) v1 from osd.2 10.252.24.30:6801/5656 -6378> 2013-04-03 10:46:43.285242 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 prepare_update pg_stats(1 pgs tid 44 v 605) v1 from osd.2 10.252.24.30:6801/5656 -6377> 2013-04-03 10:46:43.285247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 prepare_pg_stats pg_stats(1 pgs tid 44 v 605) v1 from osd.2 -6376> 2013-04-03 10:46:43.285253 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -6375> 2013-04-03 10:46:43.335442 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 encode_pending v 29130 -6374> 2013-04-03 10:46:43.335476 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427537) queue_proposal bl 654 bytes; ctx = 0x2c3dcc0 -6373> 2013-04-03 10:46:43.335486 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427537) propose_queued 427538 654 bytes -6372> 2013-04-03 10:46:43.336442 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427537 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4c80 -6371> 2013-04-03 10:46:43.364942 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114096 ==== paxos(accept lc 427537 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (950737402 0 0) 0x2fe4c80 con 0x208a840 -6370> 2013-04-03 10:46:43.367339 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427538 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2fe4000 -6369> 2013-04-03 10:46:43.367363 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427538 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250000 -6368> 2013-04-03 10:46:43.367436 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29129 update_from_paxos applying incremental 29130 -6367> 2013-04-03 10:46:43.367452 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 v29130: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2002B/s wr, 0op/s -6366> 2013-04-03 10:46:43.367472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 send_pg_creates to 0 pgs -6365> 2013-04-03 10:46:43.367479 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 update_logger -6364> 2013-04-03 10:46:43.367489 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 create_pending v 29131 -6363> 2013-04-03 10:46:43.367498 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 _updated_stats for osd.2 10.252.24.30:6801/5656 -6362> 2013-04-03 10:46:43.367506 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 44) v1 tid 17764) v2 -- ?+0 0x2b26b40 con 0x208a840 -6361> 2013-04-03 10:46:43.367517 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 check_osd_map already seen 605 >= 605 -6360> 2013-04-03 10:46:43.367521 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 update_logger -6359> 2013-04-03 10:46:43.367532 7fcdffcdb700 0 log [INF] : pgmap v29130: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2002B/s wr, 0op/s -6358> 2013-04-03 10:46:43.367534 7fcdffcdb700 10 send_log to self -6357> 2013-04-03 10:46:43.367534 7fcdffcdb700 10 log_queue is 2 last_log 196 sent 194 num 2 unsent 2 sending 2 -6356> 2013-04-03 10:46:43.367536 7fcdffcdb700 10 will send 2013-04-03 10:46:37.742673 mon.0 10.252.24.25:6789/0 195 : [INF] pgmap v29129: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 571B/s wr, 0op/s -6355> 2013-04-03 10:46:43.367542 7fcdffcdb700 10 will send 2013-04-03 10:46:43.367533 mon.0 10.252.24.25:6789/0 196 : [INF] pgmap v29130: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2002B/s wr, 0op/s -6354> 2013-04-03 10:46:43.367547 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b75600 -6353> 2013-04-03 10:46:43.367596 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos -6352> 2013-04-03 10:46:43.367616 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 -6351> 2013-04-03 10:46:43.367635 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6350> 2013-04-03 10:46:43.367656 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75600 con 0x1bac160 -6349> 2013-04-03 10:46:43.367678 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos -6348> 2013-04-03 10:46:43.367694 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 -6347> 2013-04-03 10:46:43.367703 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -6346> 2013-04-03 10:46:43.367718 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 preprocess_log log(2 entries) v1 from mon.0 -6345> 2013-04-03 10:46:43.367750 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -6344> 2013-04-03 10:46:43.367763 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 prepare_log log(2 entries) v1 from mon.0 -6343> 2013-04-03 10:46:43.367775 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 logging 2013-04-03 10:46:37.742673 mon.0 10.252.24.25:6789/0 195 : [INF] pgmap v29129: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 571B/s wr, 0op/s -6342> 2013-04-03 10:46:43.367797 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 logging 2013-04-03 10:46:43.367533 mon.0 10.252.24.25:6789/0 196 : [INF] pgmap v29130: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2002B/s wr, 0op/s -6341> 2013-04-03 10:46:43.380588 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 check_down_pgs -6340> 2013-04-03 10:46:43.380618 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 v29130: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2002B/s wr, 0op/s -6339> 2013-04-03 10:46:43.380648 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -6338> 2013-04-03 10:46:43.380670 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -6337> 2013-04-03 10:46:43.380710 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -6336> 2013-04-03 10:46:43.380741 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos -6335> 2013-04-03 10:46:43.380759 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 -6334> 2013-04-03 10:46:43.380768 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 log -6333> 2013-04-03 10:46:43.380784 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6332> 2013-04-03 10:46:43.380802 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -6331> 2013-04-03 10:46:43.389971 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114097 ==== paxos(lease_ack lc 427538 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (253022802 0 0) 0x3250000 con 0x208a840 -6330> 2013-04-03 10:46:43.390006 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos -6329> 2013-04-03 10:46:43.390023 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 update_from_paxos version 394119 summary v 394119 -6328> 2013-04-03 10:46:43.390039 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6327> 2013-04-03 10:46:43.599282 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 201 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140484 v187) v2 ==== 289+0+0 (3248448039 0 0) 0x2ce2080 con 0x2118420 -6326> 2013-04-03 10:46:43.599361 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140484 v187) v2 from mds.? 10.252.58.31:6800/10908 -6325> 2013-04-03 10:46:43.599376 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140484 v187) v2 -- ?+0 0x2ce2b80 con 0x2118420 -6324> 2013-04-03 10:46:44.365143 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394119 encode_pending v394120 -6323> 2013-04-03 10:46:44.365173 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427538) queue_proposal bl 725 bytes; ctx = 0x2c3dec0 -6322> 2013-04-03 10:46:44.365181 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427538) propose_queued 427539 725 bytes -6321> 2013-04-03 10:46:44.366272 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427538 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3250280 -6320> 2013-04-03 10:46:44.388539 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114098 ==== paxos(accept lc 427538 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3885034063 0 0) 0x3250280 con 0x208a840 -6319> 2013-04-03 10:46:44.388883 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427539 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3250000 -6318> 2013-04-03 10:46:44.388915 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427539 fc 427522 pn 0 opn 0) v3 -- ?+0 0x2fe4c80 -6317> 2013-04-03 10:46:44.388951 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos -6316> 2013-04-03 10:46:44.388979 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394119 -6315> 2013-04-03 10:46:44.388993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos latest full 393798 -6314> 2013-04-03 10:46:44.389012 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos applying incremental log 394120 2013-04-03 10:46:37.742673 mon.0 10.252.24.25:6789/0 195 : [INF] pgmap v29129: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 571B/s wr, 0op/s -6313> 2013-04-03 10:46:44.389044 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos applying incremental log 394120 2013-04-03 10:46:43.367533 mon.0 10.252.24.25:6789/0 196 : [INF] pgmap v29130: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2002B/s wr, 0op/s -6312> 2013-04-03 10:46:44.389081 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 check_subs -6311> 2013-04-03 10:46:44.389111 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 create_pending v 394121 -6310> 2013-04-03 10:46:44.389134 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 _updated_log for mon.0 10.252.24.25:6789/0 -6309> 2013-04-03 10:46:44.389148 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 196) v1 -- ?+0 0x2224380 con 0x1bac160 -6308> 2013-04-03 10:46:44.389199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos -6307> 2013-04-03 10:46:44.389216 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394120 -6306> 2013-04-03 10:46:44.389235 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6305> 2013-04-03 10:46:44.389258 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 196) v1 ==== 0+0+0 (0 0 0) 0x2224380 con 0x1bac160 -6304> 2013-04-03 10:46:44.389266 7fcdffcdb700 10 handle_log_ack log(last 196) v1 -6303> 2013-04-03 10:46:44.389268 7fcdffcdb700 10 logged 2013-04-03 10:46:37.742673 mon.0 10.252.24.25:6789/0 195 : [INF] pgmap v29129: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 571B/s wr, 0op/s -6302> 2013-04-03 10:46:44.389274 7fcdffcdb700 10 logged 2013-04-03 10:46:43.367533 mon.0 10.252.24.25:6789/0 196 : [INF] pgmap v29130: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2002B/s wr, 0op/s -6301> 2013-04-03 10:46:44.400219 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114099 ==== paxos(lease_ack lc 427539 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3538956618 0 0) 0x2fe4c80 con 0x208a840 -6300> 2013-04-03 10:46:44.400257 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos -6299> 2013-04-03 10:46:44.400274 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394120 -6298> 2013-04-03 10:46:44.400290 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6297> 2013-04-03 10:46:44.946015 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114100 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 199 v187) v2) to leader v1 ==== 560+0+0 (3652417653 0 0) 0x273a2c0 con 0x208a840 -6296> 2013-04-03 10:46:44.946037 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -6295> 2013-04-03 10:46:44.946048 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273b080 from 10.252.24.88:6789/0 -6294> 2013-04-03 10:46:44.946104 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 199 v187) v2 from mds.0 10.252.24.25:6802/3144 -6293> 2013-04-03 10:46:44.946117 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 199 v187) v2 tid 17765) v2 -- ?+0 0x24f8b40 con 0x208a840 -6292> 2013-04-03 10:46:45.397065 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 201 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21855 v187) v2 ==== 289+0+0 (2552925333 0 0) 0x2cc4840 con 0x200d420 -6291> 2013-04-03 10:46:45.397112 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21855 v187) v2 from mds.? 10.252.24.88:6800/17213 -6290> 2013-04-03 10:46:45.397122 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21855 v187) v2 -- ?+0 0x273a2c0 con 0x200d420 -6289> 2013-04-03 10:46:47.388982 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427539 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250780 -6288> 2013-04-03 10:46:47.389681 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114101 ==== paxos(lease_ack lc 427539 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1181446113 0 0) 0x3250780 con 0x208a840 -6287> 2013-04-03 10:46:47.389755 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos -6286> 2013-04-03 10:46:47.389776 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394120 -6285> 2013-04-03 10:46:47.389794 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6284> 2013-04-03 10:46:47.599386 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 202 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140485 v187) v2 ==== 289+0+0 (3807501658 0 0) 0x2ce28c0 con 0x2118420 -6283> 2013-04-03 10:46:47.599465 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140485 v187) v2 from mds.? 10.252.58.31:6800/10908 -6282> 2013-04-03 10:46:47.599480 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140485 v187) v2 -- ?+0 0x2cc4840 con 0x2118420 -6281> 2013-04-03 10:46:48.285986 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114102 ==== forward(pg_stats(1 pgs tid 45 v 605) v1) to leader v1 ==== 757+0+0 (590626052 0 0) 0x2ce2080 con 0x208a840 -6280> 2013-04-03 10:46:48.286012 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -6279> 2013-04-03 10:46:48.286024 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -6278> 2013-04-03 10:46:48.286076 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 preprocess_query pg_stats(1 pgs tid 45 v 605) v1 from osd.2 10.252.24.30:6801/5656 -6277> 2013-04-03 10:46:48.286101 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 prepare_update pg_stats(1 pgs tid 45 v 605) v1 from osd.2 10.252.24.30:6801/5656 -6276> 2013-04-03 10:46:48.286107 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 prepare_pg_stats pg_stats(1 pgs tid 45 v 605) v1 from osd.2 -6275> 2013-04-03 10:46:48.286114 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -6274> 2013-04-03 10:46:48.336307 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 encode_pending v 29131 -6273> 2013-04-03 10:46:48.336346 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427539) queue_proposal bl 654 bytes; ctx = 0x2c3dfe0 -6272> 2013-04-03 10:46:48.336356 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427539) propose_queued 427540 654 bytes -6271> 2013-04-03 10:46:48.338319 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427539 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3250500 -6270> 2013-04-03 10:46:48.362507 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114103 ==== paxos(accept lc 427539 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1768803059 0 0) 0x3250500 con 0x208a840 -6269> 2013-04-03 10:46:48.363459 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427540 fc 0 pn 17600 opn 0) v3 -- ?+0 0x1f91c80 -6268> 2013-04-03 10:46:48.363498 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427540 fc 427522 pn 0 opn 0) v3 -- ?+0 0x300ca00 -6267> 2013-04-03 10:46:48.363562 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29130 update_from_paxos applying incremental 29131 -6266> 2013-04-03 10:46:48.363578 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 v29131: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1542B/s wr, 0op/s -6265> 2013-04-03 10:46:48.363598 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 send_pg_creates to 0 pgs -6264> 2013-04-03 10:46:48.363602 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 update_logger -6263> 2013-04-03 10:46:48.363613 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 create_pending v 29132 -6262> 2013-04-03 10:46:48.363622 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 _updated_stats for osd.2 10.252.24.30:6801/5656 -6261> 2013-04-03 10:46:48.363630 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 45) v1 tid 17766) v2 -- ?+0 0x2b26b40 con 0x208a840 -6260> 2013-04-03 10:46:48.363639 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 check_osd_map already seen 605 >= 605 -6259> 2013-04-03 10:46:48.363642 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 update_logger -6258> 2013-04-03 10:46:48.363653 7fcdffcdb700 0 log [INF] : pgmap v29131: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1542B/s wr, 0op/s -6257> 2013-04-03 10:46:48.363655 7fcdffcdb700 10 send_log to self -6256> 2013-04-03 10:46:48.363655 7fcdffcdb700 10 log_queue is 1 last_log 197 sent 196 num 1 unsent 1 sending 1 -6255> 2013-04-03 10:46:48.363657 7fcdffcdb700 10 will send 2013-04-03 10:46:48.363654 mon.0 10.252.24.25:6789/0 197 : [INF] pgmap v29131: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1542B/s wr, 0op/s -6254> 2013-04-03 10:46:48.363664 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75600 -6253> 2013-04-03 10:46:48.363713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos -6252> 2013-04-03 10:46:48.363732 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394120 -6251> 2013-04-03 10:46:48.363755 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6250> 2013-04-03 10:46:48.363776 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75600 con 0x1bac160 -6249> 2013-04-03 10:46:48.363798 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos -6248> 2013-04-03 10:46:48.363813 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394120 -6247> 2013-04-03 10:46:48.363822 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6246> 2013-04-03 10:46:48.363833 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 preprocess_log log(1 entries) v1 from mon.0 -6245> 2013-04-03 10:46:48.363858 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6244> 2013-04-03 10:46:48.363871 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 prepare_log log(1 entries) v1 from mon.0 -6243> 2013-04-03 10:46:48.363881 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 logging 2013-04-03 10:46:48.363654 mon.0 10.252.24.25:6789/0 197 : [INF] pgmap v29131: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1542B/s wr, 0op/s -6242> 2013-04-03 10:46:48.380907 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 check_down_pgs -6241> 2013-04-03 10:46:48.380938 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 v29131: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1542B/s wr, 0op/s -6240> 2013-04-03 10:46:48.380966 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -6239> 2013-04-03 10:46:48.380987 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -6238> 2013-04-03 10:46:48.381024 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -6237> 2013-04-03 10:46:48.381055 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos -6236> 2013-04-03 10:46:48.381072 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394120 -6235> 2013-04-03 10:46:48.381081 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 log -6234> 2013-04-03 10:46:48.381097 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6233> 2013-04-03 10:46:48.381115 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -6232> 2013-04-03 10:46:48.385933 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114104 ==== paxos(lease_ack lc 427540 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3072931540 0 0) 0x300ca00 con 0x208a840 -6231> 2013-04-03 10:46:48.385970 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos -6230> 2013-04-03 10:46:48.385987 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 update_from_paxos version 394120 summary v 394120 -6229> 2013-04-03 10:46:48.386004 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6228> 2013-04-03 10:46:48.946084 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114105 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 200 v187) v2) to leader v1 ==== 560+0+0 (3218287901 0 0) 0x2f7a340 con 0x208a840 -6227> 2013-04-03 10:46:48.946104 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -6226> 2013-04-03 10:46:48.946115 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2f7a600 from 10.252.24.88:6789/0 -6225> 2013-04-03 10:46:48.946169 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 200 v187) v2 from mds.0 10.252.24.25:6802/3144 -6224> 2013-04-03 10:46:48.946182 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 200 v187) v2 tid 17767) v2 -- ?+0 0x24f8b40 con 0x208a840 -6223> 2013-04-03 10:46:49.362680 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394120 encode_pending v394121 -6222> 2013-04-03 10:46:49.362710 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427540) queue_proposal bl 443 bytes; ctx = 0x2c3d750 -6221> 2013-04-03 10:46:49.362718 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427540) propose_queued 427541 443 bytes -6220> 2013-04-03 10:46:49.363708 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427540 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4780 -6219> 2013-04-03 10:46:49.392903 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114106 ==== paxos(accept lc 427540 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4162792364 0 0) 0x29b4780 con 0x208a840 -6218> 2013-04-03 10:46:49.393264 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427541 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300ca00 -6217> 2013-04-03 10:46:49.393284 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427541 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250500 -6216> 2013-04-03 10:46:49.393334 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos -6215> 2013-04-03 10:46:49.393371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos version 394121 summary v 394120 -6214> 2013-04-03 10:46:49.393386 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos latest full 393798 -6213> 2013-04-03 10:46:49.393405 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos applying incremental log 394121 2013-04-03 10:46:48.363654 mon.0 10.252.24.25:6789/0 197 : [INF] pgmap v29131: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1542B/s wr, 0op/s -6212> 2013-04-03 10:46:49.393454 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 check_subs -6211> 2013-04-03 10:46:49.393486 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 create_pending v 394122 -6210> 2013-04-03 10:46:49.393509 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 _updated_log for mon.0 10.252.24.25:6789/0 -6209> 2013-04-03 10:46:49.393522 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 197) v1 -- ?+0 0x2224700 con 0x1bac160 -6208> 2013-04-03 10:46:49.393579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos -6207> 2013-04-03 10:46:49.393595 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos version 394121 summary v 394121 -6206> 2013-04-03 10:46:49.393614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6205> 2013-04-03 10:46:49.393638 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 197) v1 ==== 0+0+0 (0 0 0) 0x2224700 con 0x1bac160 -6204> 2013-04-03 10:46:49.393646 7fcdffcdb700 10 handle_log_ack log(last 197) v1 -6203> 2013-04-03 10:46:49.393648 7fcdffcdb700 10 logged 2013-04-03 10:46:48.363654 mon.0 10.252.24.25:6789/0 197 : [INF] pgmap v29131: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1542B/s wr, 0op/s -6202> 2013-04-03 10:46:49.397123 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 202 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21856 v187) v2 ==== 289+0+0 (703533993 0 0) 0x2cc42c0 con 0x200d420 -6201> 2013-04-03 10:46:49.397172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21856 v187) v2 from mds.? 10.252.24.88:6800/17213 -6200> 2013-04-03 10:46:49.397184 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21856 v187) v2 -- ?+0 0x2f7a340 con 0x200d420 -6199> 2013-04-03 10:46:49.404646 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114107 ==== paxos(lease_ack lc 427541 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (2091597555 0 0) 0x3250500 con 0x208a840 -6198> 2013-04-03 10:46:49.404701 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos -6197> 2013-04-03 10:46:49.404720 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos version 394121 summary v 394121 -6196> 2013-04-03 10:46:49.404737 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6195> 2013-04-03 10:46:51.599538 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 203 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140486 v187) v2 ==== 289+0+0 (2269256925 0 0) 0x2ce1dc0 con 0x2118420 -6194> 2013-04-03 10:46:51.599617 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140486 v187) v2 from mds.? 10.252.58.31:6800/10908 -6193> 2013-04-03 10:46:51.599632 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140486 v187) v2 -- ?+0 0x2cc42c0 con 0x2118420 -6192> 2013-04-03 10:46:51.639495 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114108 ==== forward(pg_stats(1 pgs tid 63 v 605) v1) to leader v1 ==== 753+0+0 (419350938 0 0) 0x2ce2080 con 0x208a840 -6191> 2013-04-03 10:46:51.639507 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -6190> 2013-04-03 10:46:51.639516 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -6189> 2013-04-03 10:46:51.639897 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 preprocess_query pg_stats(1 pgs tid 63 v 605) v1 from osd.1 10.252.24.25:6805/3572 -6188> 2013-04-03 10:46:51.639927 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 prepare_update pg_stats(1 pgs tid 63 v 605) v1 from osd.1 10.252.24.25:6805/3572 -6187> 2013-04-03 10:46:51.639933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 prepare_pg_stats pg_stats(1 pgs tid 63 v 605) v1 from osd.1 -6186> 2013-04-03 10:46:51.639940 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -6185> 2013-04-03 10:46:51.690131 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 encode_pending v 29132 -6184> 2013-04-03 10:46:51.690173 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427541) queue_proposal bl 650 bytes; ctx = 0x2c3de90 -6183> 2013-04-03 10:46:51.690185 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427541) propose_queued 427542 650 bytes -6182> 2013-04-03 10:46:51.691107 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427541 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4a00 -6181> 2013-04-03 10:46:51.735249 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114109 ==== paxos(accept lc 427541 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2232361385 0 0) 0x29b4a00 con 0x208a840 -6180> 2013-04-03 10:46:51.735566 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427542 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3250500 -6179> 2013-04-03 10:46:51.735596 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427542 fc 427522 pn 0 opn 0) v3 -- ?+0 0x29b4780 -6178> 2013-04-03 10:46:51.735665 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29131 update_from_paxos applying incremental 29132 -6177> 2013-04-03 10:46:51.735685 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 v29132: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6176> 2013-04-03 10:46:51.735706 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 send_pg_creates to 0 pgs -6175> 2013-04-03 10:46:51.735711 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 update_logger -6174> 2013-04-03 10:46:51.735722 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 create_pending v 29133 -6173> 2013-04-03 10:46:51.735731 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 _updated_stats for osd.1 10.252.24.25:6805/3572 -6172> 2013-04-03 10:46:51.735739 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 63) v1 tid 17768) v2 -- ?+0 0x2b26b40 con 0x208a840 -6171> 2013-04-03 10:46:51.735748 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 check_osd_map already seen 605 >= 605 -6170> 2013-04-03 10:46:51.735752 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 update_logger -6169> 2013-04-03 10:46:51.735763 7fcdffcdb700 0 log [INF] : pgmap v29132: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6168> 2013-04-03 10:46:51.735764 7fcdffcdb700 10 send_log to self -6167> 2013-04-03 10:46:51.735765 7fcdffcdb700 10 log_queue is 1 last_log 198 sent 197 num 1 unsent 1 sending 1 -6166> 2013-04-03 10:46:51.735767 7fcdffcdb700 10 will send 2013-04-03 10:46:51.735764 mon.0 10.252.24.25:6789/0 198 : [INF] pgmap v29132: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6165> 2013-04-03 10:46:51.735776 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75600 -6164> 2013-04-03 10:46:51.735825 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos -6163> 2013-04-03 10:46:51.735845 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos version 394121 summary v 394121 -6162> 2013-04-03 10:46:51.735864 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6161> 2013-04-03 10:46:51.735884 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75600 con 0x1bac160 -6160> 2013-04-03 10:46:51.735906 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos -6159> 2013-04-03 10:46:51.735921 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos version 394121 summary v 394121 -6158> 2013-04-03 10:46:51.735931 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6157> 2013-04-03 10:46:51.735945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 preprocess_log log(1 entries) v1 from mon.0 -6156> 2013-04-03 10:46:51.735968 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6155> 2013-04-03 10:46:51.735980 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 prepare_log log(1 entries) v1 from mon.0 -6154> 2013-04-03 10:46:51.735991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 logging 2013-04-03 10:46:51.735764 mon.0 10.252.24.25:6789/0 198 : [INF] pgmap v29132: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6153> 2013-04-03 10:46:51.760107 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114110 ==== paxos(lease_ack lc 427542 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (387644606 0 0) 0x29b4780 con 0x208a840 -6152> 2013-04-03 10:46:51.760184 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos -6151> 2013-04-03 10:46:51.760204 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 update_from_paxos version 394121 summary v 394121 -6150> 2013-04-03 10:46:51.760221 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6149> 2013-04-03 10:46:52.735455 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394121 encode_pending v394122 -6148> 2013-04-03 10:46:52.735484 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427542) queue_proposal bl 442 bytes; ctx = 0x2c3dea0 -6147> 2013-04-03 10:46:52.735492 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427542) propose_queued 427543 442 bytes -6146> 2013-04-03 10:46:52.737335 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427542 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4a00 -6145> 2013-04-03 10:46:52.892190 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114111 ==== paxos(accept lc 427542 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1307678968 0 0) 0x20f4a00 con 0x208a840 -6144> 2013-04-03 10:46:52.892504 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427543 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4780 -6143> 2013-04-03 10:46:52.892532 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427543 fc 427522 pn 0 opn 0) v3 -- ?+0 0x29b4a00 -6142> 2013-04-03 10:46:52.892576 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos -6141> 2013-04-03 10:46:52.892607 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394121 -6140> 2013-04-03 10:46:52.892622 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos latest full 393798 -6139> 2013-04-03 10:46:52.892643 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos applying incremental log 394122 2013-04-03 10:46:51.735764 mon.0 10.252.24.25:6789/0 198 : [INF] pgmap v29132: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6138> 2013-04-03 10:46:52.892689 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 check_subs -6137> 2013-04-03 10:46:52.892722 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 create_pending v 394123 -6136> 2013-04-03 10:46:52.892745 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 _updated_log for mon.0 10.252.24.25:6789/0 -6135> 2013-04-03 10:46:52.892759 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 198) v1 -- ?+0 0x2224e00 con 0x1bac160 -6134> 2013-04-03 10:46:52.892814 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos -6133> 2013-04-03 10:46:52.892831 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394122 -6132> 2013-04-03 10:46:52.892850 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6131> 2013-04-03 10:46:52.892874 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 198) v1 ==== 0+0+0 (0 0 0) 0x2224e00 con 0x1bac160 -6130> 2013-04-03 10:46:52.892882 7fcdffcdb700 10 handle_log_ack log(last 198) v1 -6129> 2013-04-03 10:46:52.892884 7fcdffcdb700 10 logged 2013-04-03 10:46:51.735764 mon.0 10.252.24.25:6789/0 198 : [INF] pgmap v29132: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6128> 2013-04-03 10:46:52.916933 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114112 ==== paxos(lease_ack lc 427543 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (343325609 0 0) 0x29b4a00 con 0x208a840 -6127> 2013-04-03 10:46:52.916975 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos -6126> 2013-04-03 10:46:52.916992 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394122 -6125> 2013-04-03 10:46:52.917008 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6124> 2013-04-03 10:46:52.946113 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114113 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 201 v187) v2) to leader v1 ==== 560+0+0 (1013046787 0 0) 0x2f7ab80 con 0x208a840 -6123> 2013-04-03 10:46:52.946136 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -6122> 2013-04-03 10:46:52.946147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2f7a080 from 10.252.24.88:6789/0 -6121> 2013-04-03 10:46:52.946202 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 201 v187) v2 from mds.0 10.252.24.25:6802/3144 -6120> 2013-04-03 10:46:52.946215 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 201 v187) v2 tid 17769) v2 -- ?+0 0x24f8b40 con 0x208a840 -6119> 2013-04-03 10:46:53.381237 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 check_down_pgs -6118> 2013-04-03 10:46:53.381265 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 v29132: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6117> 2013-04-03 10:46:53.381294 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -6116> 2013-04-03 10:46:53.381326 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -6115> 2013-04-03 10:46:53.381365 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -6114> 2013-04-03 10:46:53.381397 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos -6113> 2013-04-03 10:46:53.381416 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394122 -6112> 2013-04-03 10:46:53.381426 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 log -6111> 2013-04-03 10:46:53.381443 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6110> 2013-04-03 10:46:53.381463 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -6109> 2013-04-03 10:46:53.397278 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 203 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21857 v187) v2 ==== 289+0+0 (176270548 0 0) 0x2cc4000 con 0x200d420 -6108> 2013-04-03 10:46:53.397338 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21857 v187) v2 from mds.? 10.252.24.88:6800/17213 -6107> 2013-04-03 10:46:53.397351 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21857 v187) v2 -- ?+0 0x2f7ab80 con 0x200d420 -6106> 2013-04-03 10:46:55.599623 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 204 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140487 v187) v2 ==== 289+0+0 (2754388896 0 0) 0x2ce12c0 con 0x2118420 -6105> 2013-04-03 10:46:55.599709 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140487 v187) v2 from mds.? 10.252.58.31:6800/10908 -6104> 2013-04-03 10:46:55.599724 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140487 v187) v2 -- ?+0 0x2cc4000 con 0x2118420 -6103> 2013-04-03 10:46:55.892600 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427543 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3062500 -6102> 2013-04-03 10:46:55.893351 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114114 ==== paxos(lease_ack lc 427543 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1031839957 0 0) 0x3062500 con 0x208a840 -6101> 2013-04-03 10:46:55.893419 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos -6100> 2013-04-03 10:46:55.893440 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394122 -6099> 2013-04-03 10:46:55.893458 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6098> 2013-04-03 10:46:56.640120 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114115 ==== forward(pg_stats(1 pgs tid 64 v 605) v1) to leader v1 ==== 753+0+0 (1561933175 0 0) 0x2ce2080 con 0x208a840 -6097> 2013-04-03 10:46:56.640142 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -6096> 2013-04-03 10:46:56.640153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8b40 from 10.252.24.88:6789/0 -6095> 2013-04-03 10:46:56.640205 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 preprocess_query pg_stats(1 pgs tid 64 v 605) v1 from osd.1 10.252.24.25:6805/3572 -6094> 2013-04-03 10:46:56.640231 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 prepare_update pg_stats(1 pgs tid 64 v 605) v1 from osd.1 10.252.24.25:6805/3572 -6093> 2013-04-03 10:46:56.640237 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 prepare_pg_stats pg_stats(1 pgs tid 64 v 605) v1 from osd.1 -6092> 2013-04-03 10:46:56.640244 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -6091> 2013-04-03 10:46:56.690413 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 encode_pending v 29133 -6090> 2013-04-03 10:46:56.690457 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427543) queue_proposal bl 650 bytes; ctx = 0x2c3dfd0 -6089> 2013-04-03 10:46:56.690467 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427543) propose_queued 427544 650 bytes -6088> 2013-04-03 10:46:56.690873 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427543 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062280 -6087> 2013-04-03 10:46:56.719217 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114116 ==== paxos(accept lc 427543 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1829263729 0 0) 0x3062280 con 0x208a840 -6086> 2013-04-03 10:46:56.719529 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427544 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062500 -6085> 2013-04-03 10:46:56.719559 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427544 fc 427522 pn 0 opn 0) v3 -- ?+0 0x29b4a00 -6084> 2013-04-03 10:46:56.719628 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29132 update_from_paxos applying incremental 29133 -6083> 2013-04-03 10:46:56.719646 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 v29133: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6082> 2013-04-03 10:46:56.719666 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 send_pg_creates to 0 pgs -6081> 2013-04-03 10:46:56.719669 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 update_logger -6080> 2013-04-03 10:46:56.719683 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 create_pending v 29134 -6079> 2013-04-03 10:46:56.719692 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 _updated_stats for osd.1 10.252.24.25:6805/3572 -6078> 2013-04-03 10:46:56.719700 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 64) v1 tid 17770) v2 -- ?+0 0x2b26b40 con 0x208a840 -6077> 2013-04-03 10:46:56.719710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 check_osd_map already seen 605 >= 605 -6076> 2013-04-03 10:46:56.719713 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 update_logger -6075> 2013-04-03 10:46:56.719725 7fcdffcdb700 0 log [INF] : pgmap v29133: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6074> 2013-04-03 10:46:56.719727 7fcdffcdb700 10 send_log to self -6073> 2013-04-03 10:46:56.719727 7fcdffcdb700 10 log_queue is 1 last_log 199 sent 198 num 1 unsent 1 sending 1 -6072> 2013-04-03 10:46:56.719729 7fcdffcdb700 10 will send 2013-04-03 10:46:56.719726 mon.0 10.252.24.25:6789/0 199 : [INF] pgmap v29133: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6071> 2013-04-03 10:46:56.719735 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75600 -6070> 2013-04-03 10:46:56.719788 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos -6069> 2013-04-03 10:46:56.719808 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394122 -6068> 2013-04-03 10:46:56.719827 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6067> 2013-04-03 10:46:56.719849 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75600 con 0x1bac160 -6066> 2013-04-03 10:46:56.719870 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos -6065> 2013-04-03 10:46:56.719886 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394122 -6064> 2013-04-03 10:46:56.719895 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6063> 2013-04-03 10:46:56.719907 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 preprocess_log log(1 entries) v1 from mon.0 -6062> 2013-04-03 10:46:56.719933 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -6061> 2013-04-03 10:46:56.719948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 prepare_log log(1 entries) v1 from mon.0 -6060> 2013-04-03 10:46:56.719960 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 logging 2013-04-03 10:46:56.719726 mon.0 10.252.24.25:6789/0 199 : [INF] pgmap v29133: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6059> 2013-04-03 10:46:56.744092 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114117 ==== paxos(lease_ack lc 427544 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1736786803 0 0) 0x29b4a00 con 0x208a840 -6058> 2013-04-03 10:46:56.744161 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos -6057> 2013-04-03 10:46:56.744180 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 update_from_paxos version 394122 summary v 394122 -6056> 2013-04-03 10:46:56.744196 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -6055> 2013-04-03 10:46:56.805584 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114118 ==== forward(pg_stats(1 pgs tid 30463 v 605) v1) to leader v1 ==== 757+0+0 (1536723214 0 0) 0x2f79580 con 0x208a840 -6054> 2013-04-03 10:46:56.805598 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.3 10.252.24.88:6801/17538 via mon.2 10.252.24.88:6789/0 -6053> 2013-04-03 10:46:56.805607 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 -6052> 2013-04-03 10:46:56.805638 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 preprocess_query pg_stats(1 pgs tid 30463 v 605) v1 from osd.3 10.252.24.88:6801/17538 -6051> 2013-04-03 10:46:56.805659 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 prepare_update pg_stats(1 pgs tid 30463 v 605) v1 from osd.3 10.252.24.88:6801/17538 -6050> 2013-04-03 10:46:56.805665 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 prepare_pg_stats pg_stats(1 pgs tid 30463 v 605) v1 from osd.3 -6049> 2013-04-03 10:46:56.805672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -6048> 2013-04-03 10:46:56.946301 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114119 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 202 v187) v2) to leader v1 ==== 560+0+0 (2318302299 0 0) 0x2f79840 con 0x208a840 -6047> 2013-04-03 10:46:56.946316 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -6046> 2013-04-03 10:46:56.946325 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2f79000 from 10.252.24.88:6789/0 -6045> 2013-04-03 10:46:56.946359 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 202 v187) v2 from mds.0 10.252.24.25:6802/3144 -6044> 2013-04-03 10:46:56.946371 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 202 v187) v2 tid 17772) v2 -- ?+0 0x24f8b40 con 0x208a840 -6043> 2013-04-03 10:46:57.397379 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 204 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21858 v187) v2 ==== 289+0+0 (1865575763 0 0) 0x2a51b00 con 0x200d420 -6042> 2013-04-03 10:46:57.397458 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21858 v187) v2 from mds.? 10.252.24.88:6800/17213 -6041> 2013-04-03 10:46:57.397472 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21858 v187) v2 -- ?+0 0x2f79840 con 0x200d420 -6040> 2013-04-03 10:46:57.719414 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 encode_pending v 29134 -6039> 2013-04-03 10:46:57.719455 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427544) queue_proposal bl 654 bytes; ctx = 0x2c3dff0 -6038> 2013-04-03 10:46:57.719465 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427544) propose_queued 427545 654 bytes -6037> 2013-04-03 10:46:57.720572 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427544 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062780 -6036> 2013-04-03 10:46:57.720672 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394122 encode_pending v394123 -6035> 2013-04-03 10:46:57.720700 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427522..427544) queue_proposal bl 442 bytes; ctx = 0x2c3deb0 -6034> 2013-04-03 10:46:57.720707 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427522..427544) propose_new_value not active; proposal queued -6033> 2013-04-03 10:46:57.876076 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114120 ==== paxos(accept lc 427544 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3925977538 0 0) 0x3062780 con 0x208a840 -6032> 2013-04-03 10:46:57.876379 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427545 fc 0 pn 17600 opn 0) v3 -- ?+0 0x29b4a00 -6031> 2013-04-03 10:46:57.876407 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427545 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3062280 -6030> 2013-04-03 10:46:57.876478 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29133 update_from_paxos applying incremental 29134 -6029> 2013-04-03 10:46:57.876498 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 v29134: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 849B/s wr, 0op/s -6028> 2013-04-03 10:46:57.876519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 send_pg_creates to 0 pgs -6027> 2013-04-03 10:46:57.876523 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 update_logger -6026> 2013-04-03 10:46:57.876534 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 create_pending v 29135 -6025> 2013-04-03 10:46:57.876544 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 _updated_stats for osd.3 10.252.24.88:6801/17538 -6024> 2013-04-03 10:46:57.876551 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 30463) v1 tid 17771) v2 -- ?+0 0x24f8d80 con 0x208a840 -6023> 2013-04-03 10:46:57.876561 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 check_osd_map already seen 605 >= 605 -6022> 2013-04-03 10:46:57.876565 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 update_logger -6021> 2013-04-03 10:46:57.876575 7fcdffcdb700 0 log [INF] : pgmap v29134: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 849B/s wr, 0op/s -6020> 2013-04-03 10:46:57.876576 7fcdffcdb700 10 send_log to self -6019> 2013-04-03 10:46:57.876577 7fcdffcdb700 10 log_queue is 2 last_log 200 sent 199 num 2 unsent 1 sending 1 -6018> 2013-04-03 10:46:57.876579 7fcdffcdb700 10 will send 2013-04-03 10:46:56.719726 mon.0 10.252.24.25:6789/0 199 : [INF] pgmap v29133: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6017> 2013-04-03 10:46:57.876585 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74200 -6016> 2013-04-03 10:46:57.876615 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427545) propose_queued 427546 442 bytes -6015> 2013-04-03 10:46:57.876845 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427545 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4a00 -6014> 2013-04-03 10:46:57.876878 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74200 con 0x1bac160 -6013> 2013-04-03 10:46:57.887804 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114121 ==== paxos(lease_ack lc 427545 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (2076540093 0 0) 0x20f4a00 con 0x208a840 -6012> 2013-04-03 10:46:57.900806 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114122 ==== paxos(accept lc 427545 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2262690971 0 0) 0x3062280 con 0x208a840 -6011> 2013-04-03 10:46:57.901073 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427546 fc 0 pn 17600 opn 0) v3 -- ?+0 0x20f4a00 -6010> 2013-04-03 10:46:57.901105 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427546 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3062780 -6009> 2013-04-03 10:46:57.901134 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos -6008> 2013-04-03 10:46:57.901158 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos version 394123 summary v 394122 -6007> 2013-04-03 10:46:57.901172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos latest full 393798 -6006> 2013-04-03 10:46:57.901191 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos applying incremental log 394123 2013-04-03 10:46:56.719726 mon.0 10.252.24.25:6789/0 199 : [INF] pgmap v29133: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -6005> 2013-04-03 10:46:57.901441 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 check_subs -6004> 2013-04-03 10:46:57.901474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 create_pending v 394124 -6003> 2013-04-03 10:46:57.901497 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 _updated_log for mon.0 10.252.24.25:6789/0 -6002> 2013-04-03 10:46:57.901509 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 199) v1 -- ?+0 0x2224540 con 0x1bac160 -6001> 2013-04-03 10:46:57.901531 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos -6000> 2013-04-03 10:46:57.901547 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos version 394123 summary v 394123 -5999> 2013-04-03 10:46:57.901556 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -5998> 2013-04-03 10:46:57.901569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 preprocess_log log(1 entries) v1 from mon.0 -5997> 2013-04-03 10:46:57.901589 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 nothing new -5996> 2013-04-03 10:46:57.901637 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos -5995> 2013-04-03 10:46:57.901653 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos version 394123 summary v 394123 -5994> 2013-04-03 10:46:57.901672 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -5993> 2013-04-03 10:46:57.901693 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 199) v1 ==== 0+0+0 (0 0 0) 0x2224540 con 0x1bac160 -5992> 2013-04-03 10:46:57.901701 7fcdffcdb700 10 handle_log_ack log(last 199) v1 -5991> 2013-04-03 10:46:57.901702 7fcdffcdb700 10 logged 2013-04-03 10:46:56.719726 mon.0 10.252.24.25:6789/0 199 : [INF] pgmap v29133: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 980B/s wr, 0op/s -5990> 2013-04-03 10:46:57.912502 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114123 ==== paxos(lease_ack lc 427546 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (2842666763 0 0) 0x3062780 con 0x208a840 -5989> 2013-04-03 10:46:57.912542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos -5988> 2013-04-03 10:46:57.912559 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos version 394123 summary v 394123 -5987> 2013-04-03 10:46:57.912577 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -5986> 2013-04-03 10:46:58.287103 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114124 ==== forward(pg_stats(1 pgs tid 46 v 605) v1) to leader v1 ==== 757+0+0 (3662771139 0 0) 0x2f79580 con 0x208a840 -5985> 2013-04-03 10:46:58.287130 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -5984> 2013-04-03 10:46:58.287141 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -5983> 2013-04-03 10:46:58.287191 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 preprocess_query pg_stats(1 pgs tid 46 v 605) v1 from osd.2 10.252.24.30:6801/5656 -5982> 2013-04-03 10:46:58.287216 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 prepare_update pg_stats(1 pgs tid 46 v 605) v1 from osd.2 10.252.24.30:6801/5656 -5981> 2013-04-03 10:46:58.287223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 prepare_pg_stats pg_stats(1 pgs tid 46 v 605) v1 from osd.2 -5980> 2013-04-03 10:46:58.287229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -5979> 2013-04-03 10:46:58.381587 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 check_down_pgs -5978> 2013-04-03 10:46:58.381619 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 v29134: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 849B/s wr, 0op/s -5977> 2013-04-03 10:46:58.381649 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -5976> 2013-04-03 10:46:58.381672 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 e605: 5 osds: 5 up, 5 in -5975> 2013-04-03 10:46:58.381710 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e605 min_last_epoch_clean 593 -5974> 2013-04-03 10:46:58.381741 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos -5973> 2013-04-03 10:46:58.381761 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 update_from_paxos version 394123 summary v 394123 -5972> 2013-04-03 10:46:58.381773 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394123 log -5971> 2013-04-03 10:46:58.381791 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 update_from_paxos -5970> 2013-04-03 10:46:58.381810 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2061 auth -5969> 2013-04-03 10:46:58.901001 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29134 encode_pending v 29135 -5968> 2013-04-03 10:46:58.901039 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427546) queue_proposal bl 654 bytes; ctx = 0x2c3de80 -5967> 2013-04-03 10:46:58.901048 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427546) propose_queued 427547 654 bytes -5966> 2013-04-03 10:46:58.902404 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427546 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003280 -5965> 2013-04-03 10:46:58.927688 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) -5964> 2013-04-03 10:46:58.927703 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader -5963> 2013-04-03 10:46:58.927981 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114125 ==== paxos(accept lc 427546 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3352490585 0 0) 0x3003280 con 0x208a840 -5962> 2013-04-03 10:46:58.928298 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427547 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062780 -5961> 2013-04-03 10:46:58.928316 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427547 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3062280 -5960> 2013-04-03 10:46:58.928390 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29134 update_from_paxos applying incremental 29135 -5959> 2013-04-03 10:46:58.928410 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 v29135: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 33815B/s wr, 0op/s -5958> 2013-04-03 10:46:58.928430 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 send_pg_creates to 0 pgs -5957> 2013-04-03 10:46:58.928432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 update_logger -5956> 2013-04-03 10:46:58.928442 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 create_pending v 29136 -5955> 2013-04-03 10:46:58.928451 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29135 _updated_stats for osd.2 10.252.24.30:6801/5656 -5954> 2013-04-03 10:46:58.928457 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 46) v1 tid 17773) v2 -- ?+0 0x2b26b40 con 0x208a840 -5953> 2013-04-03 10:46:58.928468 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 check_osd_map already seen 605 >= 605 -5952> 2013-04-03 10:46:58.928471 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 update_logger -5951> 2013-04-03 10:46:58.928481 7fcdffcdb700 0 log [INF] : pgmap v29135: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 33815B/s wr, 0op/s -5950> 2013-04-03 10:46:58.928483 7fcdffcdb700 10 send_log to self -5949> 2013-04-03 10:46:58.928484 7fcdffcdb700 10 log_queue is 2 last_log 201 sent 199 num 2 unsent 2 sending 2 -5948> 2013-04-03 10:46:58.928485 7fcdffcdb700 10 will send 2013-04-03 10:46:57.876576 mon.0 10.252.24.25:6789/0 200 : [INF] pgmap v29134: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 849B/s wr, 0op/s -5947> 2013-04-03 10:46:58.928491 7fcdffcdb700 10 will send 2013-04-03 10:46:58.928482 mon.0 10.252.24.25:6789/0 201 : [INF] pgmap v29135: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 33815B/s wr, 0op/s -5946> 2013-04-03 10:46:58.928502 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b74200 -5945> 2013-04-03 10:46:58.928556 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 update_from_paxos -5944> 2013-04-03 10:46:58.928575 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 update_from_paxos version 394123 summary v 394123 -5943> 2013-04-03 10:46:58.928595 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5942> 2013-04-03 10:46:58.928618 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74200 con 0x1bac160 -5941> 2013-04-03 10:46:58.928640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 update_from_paxos -5940> 2013-04-03 10:46:58.928655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 update_from_paxos version 394123 summary v 394123 -5939> 2013-04-03 10:46:58.928664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -5938> 2013-04-03 10:46:58.928676 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 preprocess_log log(2 entries) v1 from mon.0 -5937> 2013-04-03 10:46:58.928708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -5936> 2013-04-03 10:46:58.928720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 prepare_log log(2 entries) v1 from mon.0 -5935> 2013-04-03 10:46:58.928731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 logging 2013-04-03 10:46:57.876576 mon.0 10.252.24.25:6789/0 200 : [INF] pgmap v29134: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 849B/s wr, 0op/s -5934> 2013-04-03 10:46:58.928752 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 logging 2013-04-03 10:46:58.928482 mon.0 10.252.24.25:6789/0 201 : [INF] pgmap v29135: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 33815B/s wr, 0op/s -5933> 2013-04-03 10:46:58.939915 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114126 ==== paxos(lease_ack lc 427547 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (4023483391 0 0) 0x3062280 con 0x208a840 -5932> 2013-04-03 10:46:58.939950 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 update_from_paxos -5931> 2013-04-03 10:46:58.939966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394123 update_from_paxos version 394123 summary v 394123 -5930> 2013-04-03 10:46:58.939983 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5929> 2013-04-03 10:46:59.599772 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 205 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140488 v187) v2 ==== 289+0+0 (1388057290 0 0) 0x273a000 con 0x2118420 -5928> 2013-04-03 10:46:59.599849 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140488 v187) v2 from mds.? 10.252.58.31:6800/10908 -5927> 2013-04-03 10:46:59.599863 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140488 v187) v2 -- ?+0 0x2f79580 con 0x2118420 -5926> 2013-04-03 10:46:59.928178 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394123 encode_pending v394124 -5925> 2013-04-03 10:46:59.928206 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427547) queue_proposal bl 726 bytes; ctx = 0x2c3dfb0 -5924> 2013-04-03 10:46:59.928212 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427547) propose_queued 427548 726 bytes -5923> 2013-04-03 10:46:59.929325 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427547 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3003000 -5922> 2013-04-03 10:46:59.979973 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114127 ==== paxos(accept lc 427547 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (1834552354 0 0) 0x3003000 con 0x208a840 -5921> 2013-04-03 10:46:59.980268 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427548 fc 0 pn 17600 opn 0) v3 -- ?+0 0x3062280 -5920> 2013-04-03 10:46:59.980290 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427548 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3003280 -5919> 2013-04-03 10:46:59.980333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos -5918> 2013-04-03 10:46:59.980360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394123 -5917> 2013-04-03 10:46:59.980374 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos latest full 393798 -5916> 2013-04-03 10:46:59.980394 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394124 update_from_paxos applying incremental log 394124 2013-04-03 10:46:57.876576 mon.0 10.252.24.25:6789/0 200 : [INF] pgmap v29134: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 849B/s wr, 0op/s -5915> 2013-04-03 10:46:59.980422 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394124 update_from_paxos applying incremental log 394124 2013-04-03 10:46:58.928482 mon.0 10.252.24.25:6789/0 201 : [INF] pgmap v29135: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 33815B/s wr, 0op/s -5914> 2013-04-03 10:46:59.980464 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 check_subs -5913> 2013-04-03 10:46:59.980496 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 create_pending v 394125 -5912> 2013-04-03 10:46:59.980518 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394124 _updated_log for mon.0 10.252.24.25:6789/0 -5911> 2013-04-03 10:46:59.980531 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 201) v1 -- ?+0 0x2224000 con 0x1bac160 -5910> 2013-04-03 10:46:59.980582 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos -5909> 2013-04-03 10:46:59.980597 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 -5908> 2013-04-03 10:46:59.980615 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5907> 2013-04-03 10:46:59.980638 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 201) v1 ==== 0+0+0 (0 0 0) 0x2224000 con 0x1bac160 -5906> 2013-04-03 10:46:59.980646 7fcdffcdb700 10 handle_log_ack log(last 201) v1 -5905> 2013-04-03 10:46:59.980647 7fcdffcdb700 10 logged 2013-04-03 10:46:57.876576 mon.0 10.252.24.25:6789/0 200 : [INF] pgmap v29134: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 849B/s wr, 0op/s -5904> 2013-04-03 10:46:59.980653 7fcdffcdb700 10 logged 2013-04-03 10:46:58.928482 mon.0 10.252.24.25:6789/0 201 : [INF] pgmap v29135: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 33815B/s wr, 0op/s -5903> 2013-04-03 10:47:00.004821 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114128 ==== paxos(lease_ack lc 427548 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (289377035 0 0) 0x3003280 con 0x208a840 -5902> 2013-04-03 10:47:00.004897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos -5901> 2013-04-03 10:47:00.004919 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 -5900> 2013-04-03 10:47:00.004935 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5899> 2013-04-03 10:47:00.946482 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114129 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 203 v187) v2) to leader v1 ==== 560+0+0 (1465612558 0 0) 0x2a51840 con 0x208a840 -5898> 2013-04-03 10:47:00.946504 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -5897> 2013-04-03 10:47:00.946514 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a512c0 from 10.252.24.88:6789/0 -5896> 2013-04-03 10:47:00.946568 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 203 v187) v2 from mds.0 10.252.24.25:6802/3144 -5895> 2013-04-03 10:47:00.946579 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 203 v187) v2 tid 17774) v2 -- ?+0 0x24f8000 con 0x208a840 -5894> 2013-04-03 10:47:01.305199 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x300cc80 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33179/0 -5893> 2013-04-03 10:47:01.305226 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x300cc80 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33179/0) -5892> 2013-04-03 10:47:01.305239 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x300cc80 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -5891> 2013-04-03 10:47:01.305441 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x300cc80 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -5890> 2013-04-03 10:47:01.397488 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 205 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21859 v187) v2 ==== 289+0+0 (1281151534 0 0) 0x2a528c0 con 0x200d420 -5889> 2013-04-03 10:47:01.397556 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21859 v187) v2 from mds.? 10.252.24.88:6800/17213 -5888> 2013-04-03 10:47:01.397569 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21859 v187) v2 -- ?+0 0x2a51840 con 0x200d420 -5887> 2013-04-03 10:47:02.980367 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427548 fc 427522 pn 0 opn 0) v3 -- ?+0 0x2116a00 -5886> 2013-04-03 10:47:02.981153 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114130 ==== paxos(lease_ack lc 427548 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (2043072417 0 0) 0x2116a00 con 0x208a840 -5885> 2013-04-03 10:47:02.981221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos -5884> 2013-04-03 10:47:02.981240 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 -5883> 2013-04-03 10:47:02.981258 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5882> 2013-04-03 10:47:03.381928 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29135 check_down_pgs -5881> 2013-04-03 10:47:03.381958 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29135 v29135: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 33815B/s wr, 0op/s -5880> 2013-04-03 10:47:03.381986 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -5879> 2013-04-03 10:47:03.382010 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -5878> 2013-04-03 10:47:03.382047 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -5877> 2013-04-03 10:47:03.382082 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos -5876> 2013-04-03 10:47:03.382100 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 -5875> 2013-04-03 10:47:03.382109 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394124 log -5874> 2013-04-03 10:47:03.382126 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5873> 2013-04-03 10:47:03.382145 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -5872> 2013-04-03 10:47:03.599898 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 206 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140489 v187) v2 ==== 289+0+0 (1909632439 0 0) 0x273bb80 con 0x2118420 -5871> 2013-04-03 10:47:03.599977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140489 v187) v2 from mds.? 10.252.58.31:6800/10908 -5870> 2013-04-03 10:47:03.599991 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140489 v187) v2 -- ?+0 0x2a528c0 con 0x2118420 -5869> 2013-04-03 10:47:04.946670 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114131 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 204 v187) v2) to leader v1 ==== 560+0+0 (3291718490 0 0) 0x273a000 con 0x208a840 -5868> 2013-04-03 10:47:04.946692 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -5867> 2013-04-03 10:47:04.946705 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52b80 from 10.252.24.88:6789/0 -5866> 2013-04-03 10:47:04.946759 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 204 v187) v2 from mds.0 10.252.24.25:6802/3144 -5865> 2013-04-03 10:47:04.946770 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 204 v187) v2 tid 17775) v2 -- ?+0 0x2b26b40 con 0x208a840 -5864> 2013-04-03 10:47:05.397585 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 206 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21860 v187) v2 ==== 289+0+0 (2757095005 0 0) 0x2a52080 con 0x200d420 -5863> 2013-04-03 10:47:05.397656 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21860 v187) v2 from mds.? 10.252.24.88:6800/17213 -5862> 2013-04-03 10:47:05.397669 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21860 v187) v2 -- ?+0 0x273a000 con 0x200d420 -5861> 2013-04-03 10:47:05.980439 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427548 fc 427522 pn 0 opn 0) v3 -- ?+0 0x2116000 -5860> 2013-04-03 10:47:05.981204 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114132 ==== paxos(lease_ack lc 427548 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1108596490 0 0) 0x2116000 con 0x208a840 -5859> 2013-04-03 10:47:05.981277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos -5858> 2013-04-03 10:47:05.981297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 -5857> 2013-04-03 10:47:05.981318 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5856> 2013-04-03 10:47:06.641403 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114133 ==== forward(pg_stats(1 pgs tid 65 v 605) v1) to leader v1 ==== 753+0+0 (2453342454 0 0) 0x273bb80 con 0x208a840 -5855> 2013-04-03 10:47:06.641423 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.1 10.252.24.25:6805/3572 via mon.2 10.252.24.88:6789/0 -5854> 2013-04-03 10:47:06.641432 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2b26b40 from 10.252.24.88:6789/0 -5853> 2013-04-03 10:47:06.641485 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 preprocess_query pg_stats(1 pgs tid 65 v 605) v1 from osd.1 10.252.24.25:6805/3572 -5852> 2013-04-03 10:47:06.641721 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 prepare_update pg_stats(1 pgs tid 65 v 605) v1 from osd.1 10.252.24.25:6805/3572 -5851> 2013-04-03 10:47:06.641732 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 prepare_pg_stats pg_stats(1 pgs tid 65 v 605) v1 from osd.1 -5850> 2013-04-03 10:47:06.641738 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29135 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -5849> 2013-04-03 10:47:06.691924 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29135 encode_pending v 29136 -5848> 2013-04-03 10:47:06.691959 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427548) queue_proposal bl 650 bytes; ctx = 0x2c3dfa0 -5847> 2013-04-03 10:47:06.691968 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427548) propose_queued 427549 650 bytes -5846> 2013-04-03 10:47:06.693061 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427548 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300c280 -5845> 2013-04-03 10:47:06.718819 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114134 ==== paxos(accept lc 427548 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2087607160 0 0) 0x300c280 con 0x208a840 -5844> 2013-04-03 10:47:06.719098 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427549 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116000 -5843> 2013-04-03 10:47:06.719129 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427549 fc 427522 pn 0 opn 0) v3 -- ?+0 0x2116a00 -5842> 2013-04-03 10:47:06.719201 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29135 update_from_paxos applying incremental 29136 -5841> 2013-04-03 10:47:06.719217 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 v29136: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 8673B/s wr, 0op/s -5840> 2013-04-03 10:47:06.719236 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 send_pg_creates to 0 pgs -5839> 2013-04-03 10:47:06.719239 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 update_logger -5838> 2013-04-03 10:47:06.719249 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 create_pending v 29137 -5837> 2013-04-03 10:47:06.719258 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29136 _updated_stats for osd.1 10.252.24.25:6805/3572 -5836> 2013-04-03 10:47:06.719265 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 65) v1 tid 17776) v2 -- ?+0 0x24f8000 con 0x208a840 -5835> 2013-04-03 10:47:06.719274 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 check_osd_map already seen 605 >= 605 -5834> 2013-04-03 10:47:06.719277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 update_logger -5833> 2013-04-03 10:47:06.719287 7fcdffcdb700 0 log [INF] : pgmap v29136: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 8673B/s wr, 0op/s -5832> 2013-04-03 10:47:06.719289 7fcdffcdb700 10 send_log to self -5831> 2013-04-03 10:47:06.719290 7fcdffcdb700 10 log_queue is 1 last_log 202 sent 201 num 1 unsent 1 sending 1 -5830> 2013-04-03 10:47:06.719291 7fcdffcdb700 10 will send 2013-04-03 10:47:06.719288 mon.0 10.252.24.25:6789/0 202 : [INF] pgmap v29136: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 8673B/s wr, 0op/s -5829> 2013-04-03 10:47:06.719298 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74200 -5828> 2013-04-03 10:47:06.719346 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos -5827> 2013-04-03 10:47:06.719365 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 -5826> 2013-04-03 10:47:06.719383 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5825> 2013-04-03 10:47:06.719404 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74200 con 0x1bac160 -5824> 2013-04-03 10:47:06.719428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos -5823> 2013-04-03 10:47:06.719443 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 -5822> 2013-04-03 10:47:06.719452 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -5821> 2013-04-03 10:47:06.719464 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 preprocess_log log(1 entries) v1 from mon.0 -5820> 2013-04-03 10:47:06.719487 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -5819> 2013-04-03 10:47:06.719499 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 prepare_log log(1 entries) v1 from mon.0 -5818> 2013-04-03 10:47:06.719509 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 logging 2013-04-03 10:47:06.719288 mon.0 10.252.24.25:6789/0 202 : [INF] pgmap v29136: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 8673B/s wr, 0op/s -5817> 2013-04-03 10:47:06.731860 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114135 ==== paxos(lease_ack lc 427549 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (2082146574 0 0) 0x2116a00 con 0x208a840 -5816> 2013-04-03 10:47:06.731905 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos -5815> 2013-04-03 10:47:06.731922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394124 update_from_paxos version 394124 summary v 394124 -5814> 2013-04-03 10:47:06.731937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5813> 2013-04-03 10:47:07.600007 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 207 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140490 v187) v2 ==== 289+0+0 (341929008 0 0) 0x273b340 con 0x2118420 -5812> 2013-04-03 10:47:07.600083 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140490 v187) v2 from mds.? 10.252.58.31:6800/10908 -5811> 2013-04-03 10:47:07.600096 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140490 v187) v2 -- ?+0 0x273bb80 con 0x2118420 -5810> 2013-04-03 10:47:07.719009 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394124 encode_pending v394125 -5809> 2013-04-03 10:47:07.719035 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427549) queue_proposal bl 443 bytes; ctx = 0x2c3de50 -5808> 2013-04-03 10:47:07.719042 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427549) propose_queued 427550 443 bytes -5807> 2013-04-03 10:47:07.720590 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427549 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300cc80 -5806> 2013-04-03 10:47:07.912532 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114136 ==== paxos(accept lc 427549 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (3951259764 0 0) 0x300cc80 con 0x208a840 -5805> 2013-04-03 10:47:07.912825 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427550 fc 0 pn 17600 opn 0) v3 -- ?+0 0x2116a00 -5804> 2013-04-03 10:47:07.912858 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427550 fc 427522 pn 0 opn 0) v3 -- ?+0 0x300c280 -5803> 2013-04-03 10:47:07.912896 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos -5802> 2013-04-03 10:47:07.912922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394124 -5801> 2013-04-03 10:47:07.912939 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos latest full 393798 -5800> 2013-04-03 10:47:07.912956 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394125 update_from_paxos applying incremental log 394125 2013-04-03 10:47:06.719288 mon.0 10.252.24.25:6789/0 202 : [INF] pgmap v29136: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 8673B/s wr, 0op/s -5799> 2013-04-03 10:47:07.913000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 check_subs -5798> 2013-04-03 10:47:07.913032 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 create_pending v 394126 -5797> 2013-04-03 10:47:07.913054 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394125 _updated_log for mon.0 10.252.24.25:6789/0 -5796> 2013-04-03 10:47:07.913066 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 202) v1 -- ?+0 0x22241c0 con 0x1bac160 -5795> 2013-04-03 10:47:07.913115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos -5794> 2013-04-03 10:47:07.913131 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 -5793> 2013-04-03 10:47:07.913147 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5792> 2013-04-03 10:47:07.913168 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 202) v1 ==== 0+0+0 (0 0 0) 0x22241c0 con 0x1bac160 -5791> 2013-04-03 10:47:07.913175 7fcdffcdb700 10 handle_log_ack log(last 202) v1 -5790> 2013-04-03 10:47:07.913177 7fcdffcdb700 10 logged 2013-04-03 10:47:06.719288 mon.0 10.252.24.25:6789/0 202 : [INF] pgmap v29136: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 8673B/s wr, 0op/s -5789> 2013-04-03 10:47:07.925541 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114137 ==== paxos(lease_ack lc 427550 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1681530865 0 0) 0x300c280 con 0x208a840 -5788> 2013-04-03 10:47:07.925580 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos -5787> 2013-04-03 10:47:07.925597 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 -5786> 2013-04-03 10:47:07.925611 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5785> 2013-04-03 10:47:08.382285 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29136 check_down_pgs -5784> 2013-04-03 10:47:08.382322 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29136 v29136: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 8673B/s wr, 0op/s -5783> 2013-04-03 10:47:08.382351 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -5782> 2013-04-03 10:47:08.382373 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -5781> 2013-04-03 10:47:08.382410 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -5780> 2013-04-03 10:47:08.382441 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos -5779> 2013-04-03 10:47:08.382459 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 -5778> 2013-04-03 10:47:08.382469 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394125 log -5777> 2013-04-03 10:47:08.382486 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5776> 2013-04-03 10:47:08.382505 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -5775> 2013-04-03 10:47:08.946787 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114138 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 205 v187) v2) to leader v1 ==== 560+0+0 (2440304800 0 0) 0x2a51000 con 0x208a840 -5774> 2013-04-03 10:47:08.946808 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -5773> 2013-04-03 10:47:08.946818 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a51dc0 from 10.252.24.88:6789/0 -5772> 2013-04-03 10:47:08.946875 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 205 v187) v2 from mds.0 10.252.24.25:6802/3144 -5771> 2013-04-03 10:47:08.946886 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 205 v187) v2 tid 17777) v2 -- ?+0 0x2b26b40 con 0x208a840 -5770> 2013-04-03 10:47:09.397676 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 207 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21861 v187) v2 ==== 289+0+0 (2268811552 0 0) 0x2a52600 con 0x200d420 -5769> 2013-04-03 10:47:09.397747 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21861 v187) v2 from mds.? 10.252.24.88:6800/17213 -5768> 2013-04-03 10:47:09.397759 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21861 v187) v2 -- ?+0 0x2a51000 con 0x200d420 -5767> 2013-04-03 10:47:10.912944 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427550 fc 427522 pn 0 opn 0) v3 -- ?+0 0x32dfc80 -5766> 2013-04-03 10:47:10.913650 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114139 ==== paxos(lease_ack lc 427550 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1862832690 0 0) 0x32dfc80 con 0x208a840 -5765> 2013-04-03 10:47:10.913724 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos -5764> 2013-04-03 10:47:10.913744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 -5763> 2013-04-03 10:47:10.913761 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5762> 2013-04-03 10:47:11.321194 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x300c000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33180/0 -5761> 2013-04-03 10:47:11.321220 7fcdfe7d4700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x300c000 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33180/0) -5760> 2013-04-03 10:47:11.321233 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x300c000 sd=25 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -5759> 2013-04-03 10:47:11.321427 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x300c000 sd=25 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -5758> 2013-04-03 10:47:11.600147 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 208 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140491 v187) v2 ==== 289+0+0 (923784013 0 0) 0x273b600 con 0x2118420 -5757> 2013-04-03 10:47:11.600225 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140491 v187) v2 from mds.? 10.252.58.31:6800/10908 -5756> 2013-04-03 10:47:11.600238 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140491 v187) v2 -- ?+0 0x2a52600 con 0x2118420 -5755> 2013-04-03 10:47:12.946943 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114140 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 206 v187) v2) to leader v1 ==== 560+0+0 (2326690469 0 0) 0x273b340 con 0x208a840 -5754> 2013-04-03 10:47:12.946965 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -5753> 2013-04-03 10:47:12.946975 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x21452c0 from 10.252.24.88:6789/0 -5752> 2013-04-03 10:47:12.947029 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 206 v187) v2 from mds.0 10.252.24.25:6802/3144 -5751> 2013-04-03 10:47:12.947039 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 206 v187) v2 tid 17778) v2 -- ?+0 0x24f8000 con 0x208a840 -5750> 2013-04-03 10:47:13.289113 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114141 ==== forward(pg_stats(1 pgs tid 47 v 605) v1) to leader v1 ==== 757+0+0 (2389944619 0 0) 0x273b600 con 0x208a840 -5749> 2013-04-03 10:47:13.289141 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -5748> 2013-04-03 10:47:13.289153 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 -5747> 2013-04-03 10:47:13.289204 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 preprocess_query pg_stats(1 pgs tid 47 v 605) v1 from osd.2 10.252.24.30:6801/5656 -5746> 2013-04-03 10:47:13.289228 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 prepare_update pg_stats(1 pgs tid 47 v 605) v1 from osd.2 10.252.24.30:6801/5656 -5745> 2013-04-03 10:47:13.289234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 prepare_pg_stats pg_stats(1 pgs tid 47 v 605) v1 from osd.2 -5744> 2013-04-03 10:47:13.289240 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29136 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -5743> 2013-04-03 10:47:13.339421 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29136 encode_pending v 29137 -5742> 2013-04-03 10:47:13.339452 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427550) queue_proposal bl 654 bytes; ctx = 0x2c3df90 -5741> 2013-04-03 10:47:13.339460 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427550) propose_queued 427551 654 bytes -5740> 2013-04-03 10:47:13.340564 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427550 fc 0 pn 17600 opn 0) v3 -- ?+0 0x32dfa00 -5739> 2013-04-03 10:47:13.368389 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114142 ==== paxos(accept lc 427550 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (2949339407 0 0) 0x32dfa00 con 0x208a840 -5738> 2013-04-03 10:47:13.368697 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427551 fc 0 pn 17600 opn 0) v3 -- ?+0 0x32dfc80 -5737> 2013-04-03 10:47:13.368727 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427551 fc 427522 pn 0 opn 0) v3 -- ?+0 0x300c280 -5736> 2013-04-03 10:47:13.368795 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29136 update_from_paxos applying incremental 29137 -5735> 2013-04-03 10:47:13.368810 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 v29137: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1985B/s wr, 0op/s -5734> 2013-04-03 10:47:13.368828 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 send_pg_creates to 0 pgs -5733> 2013-04-03 10:47:13.368830 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 update_logger -5732> 2013-04-03 10:47:13.369115 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 create_pending v 29138 -5731> 2013-04-03 10:47:13.369131 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29137 _updated_stats for osd.2 10.252.24.30:6801/5656 -5730> 2013-04-03 10:47:13.369138 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 47) v1 tid 17779) v2 -- ?+0 0x2b26b40 con 0x208a840 -5729> 2013-04-03 10:47:13.369149 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 check_osd_map already seen 605 >= 605 -5728> 2013-04-03 10:47:13.369152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 update_logger -5727> 2013-04-03 10:47:13.369163 7fcdffcdb700 0 log [INF] : pgmap v29137: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1985B/s wr, 0op/s -5726> 2013-04-03 10:47:13.369164 7fcdffcdb700 10 send_log to self -5725> 2013-04-03 10:47:13.369165 7fcdffcdb700 10 log_queue is 1 last_log 203 sent 202 num 1 unsent 1 sending 1 -5724> 2013-04-03 10:47:13.369167 7fcdffcdb700 10 will send 2013-04-03 10:47:13.369164 mon.0 10.252.24.25:6789/0 203 : [INF] pgmap v29137: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1985B/s wr, 0op/s -5723> 2013-04-03 10:47:13.369174 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74200 -5722> 2013-04-03 10:47:13.369223 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos -5721> 2013-04-03 10:47:13.369243 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 -5720> 2013-04-03 10:47:13.369259 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5719> 2013-04-03 10:47:13.369280 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74200 con 0x1bac160 -5718> 2013-04-03 10:47:13.369300 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos -5717> 2013-04-03 10:47:13.369315 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 -5716> 2013-04-03 10:47:13.369323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -5715> 2013-04-03 10:47:13.369333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 preprocess_log log(1 entries) v1 from mon.0 -5714> 2013-04-03 10:47:13.369355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -5713> 2013-04-03 10:47:13.369367 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 prepare_log log(1 entries) v1 from mon.0 -5712> 2013-04-03 10:47:13.369376 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 logging 2013-04-03 10:47:13.369164 mon.0 10.252.24.25:6789/0 203 : [INF] pgmap v29137: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1985B/s wr, 0op/s -5711> 2013-04-03 10:47:13.381383 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114143 ==== paxos(lease_ack lc 427551 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1753426405 0 0) 0x300c280 con 0x208a840 -5710> 2013-04-03 10:47:13.381424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos -5709> 2013-04-03 10:47:13.381441 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 -5708> 2013-04-03 10:47:13.381456 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5707> 2013-04-03 10:47:13.382617 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29137 check_down_pgs -5706> 2013-04-03 10:47:13.382646 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29137 v29137: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1985B/s wr, 0op/s -5705> 2013-04-03 10:47:13.382674 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -5704> 2013-04-03 10:47:13.382695 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -5703> 2013-04-03 10:47:13.382732 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -5702> 2013-04-03 10:47:13.382765 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos -5701> 2013-04-03 10:47:13.382789 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394125 update_from_paxos version 394125 summary v 394125 -5700> 2013-04-03 10:47:13.382797 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394125 log -5699> 2013-04-03 10:47:13.382812 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5698> 2013-04-03 10:47:13.382828 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -5697> 2013-04-03 10:47:13.397749 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 208 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21862 v187) v2 ==== 289+0+0 (3800600743 0 0) 0x2145840 con 0x200d420 -5696> 2013-04-03 10:47:13.397802 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21862 v187) v2 from mds.? 10.252.24.88:6800/17213 -5695> 2013-04-03 10:47:13.397818 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21862 v187) v2 -- ?+0 0x273b600 con 0x200d420 -5694> 2013-04-03 10:47:14.368596 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394125 encode_pending v394126 -5693> 2013-04-03 10:47:14.368630 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427551) queue_proposal bl 443 bytes; ctx = 0x2c3de40 -5692> 2013-04-03 10:47:14.368638 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427551) propose_queued 427552 443 bytes -5691> 2013-04-03 10:47:14.369722 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427551 fc 0 pn 17600 opn 0) v3 -- ?+0 0x32df780 -5690> 2013-04-03 10:47:14.396884 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114144 ==== paxos(accept lc 427551 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (4225170218 0 0) 0x32df780 con 0x208a840 -5689> 2013-04-03 10:47:14.397196 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427552 fc 0 pn 17600 opn 0) v3 -- ?+0 0x300c280 -5688> 2013-04-03 10:47:14.397230 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427552 fc 427522 pn 0 opn 0) v3 -- ?+0 0x32dfa00 -5687> 2013-04-03 10:47:14.397265 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos -5686> 2013-04-03 10:47:14.397292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394125 -5685> 2013-04-03 10:47:14.397307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos latest full 393798 -5684> 2013-04-03 10:47:14.397323 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394126 update_from_paxos applying incremental log 394126 2013-04-03 10:47:13.369164 mon.0 10.252.24.25:6789/0 203 : [INF] pgmap v29137: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1985B/s wr, 0op/s -5683> 2013-04-03 10:47:14.397371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 check_subs -5682> 2013-04-03 10:47:14.397403 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 create_pending v 394127 -5681> 2013-04-03 10:47:14.397425 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394126 _updated_log for mon.0 10.252.24.25:6789/0 -5680> 2013-04-03 10:47:14.397438 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 203) v1 -- ?+0 0x2224c40 con 0x1bac160 -5679> 2013-04-03 10:47:14.397489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos -5678> 2013-04-03 10:47:14.397505 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394126 -5677> 2013-04-03 10:47:14.397526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5676> 2013-04-03 10:47:14.397549 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 203) v1 ==== 0+0+0 (0 0 0) 0x2224c40 con 0x1bac160 -5675> 2013-04-03 10:47:14.397558 7fcdffcdb700 10 handle_log_ack log(last 203) v1 -5674> 2013-04-03 10:47:14.397559 7fcdffcdb700 10 logged 2013-04-03 10:47:13.369164 mon.0 10.252.24.25:6789/0 203 : [INF] pgmap v29137: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 1985B/s wr, 0op/s -5673> 2013-04-03 10:47:14.408592 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114145 ==== paxos(lease_ack lc 427552 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (984547336 0 0) 0x32dfa00 con 0x208a840 -5672> 2013-04-03 10:47:14.408633 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos -5671> 2013-04-03 10:47:14.408650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394126 -5670> 2013-04-03 10:47:14.408665 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5669> 2013-04-03 10:47:15.600231 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 209 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140492 v187) v2 ==== 289+0+0 (3741773630 0 0) 0x273adc0 con 0x2118420 -5668> 2013-04-03 10:47:15.600314 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140492 v187) v2 from mds.? 10.252.58.31:6800/10908 -5667> 2013-04-03 10:47:15.600327 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140492 v187) v2 -- ?+0 0x2145840 con 0x2118420 -5666> 2013-04-03 10:47:16.947028 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114146 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 207 v187) v2) to leader v1 ==== 560+0+0 (1650863561 0 0) 0x2145580 con 0x208a840 -5665> 2013-04-03 10:47:16.947050 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -5664> 2013-04-03 10:47:16.947060 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2145b00 from 10.252.24.88:6789/0 -5663> 2013-04-03 10:47:16.947114 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 207 v187) v2 from mds.0 10.252.24.25:6802/3144 -5662> 2013-04-03 10:47:16.947126 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 207 v187) v2 tid 17780) v2 -- ?+0 0x24f8000 con 0x208a840 -5661> 2013-04-03 10:47:17.397306 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427552 fc 427522 pn 0 opn 0) v3 -- ?+0 0x32df500 -5660> 2013-04-03 10:47:17.397864 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 209 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21863 v187) v2 ==== 289+0+0 (3253088218 0 0) 0x2146080 con 0x200d420 -5659> 2013-04-03 10:47:17.397942 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21863 v187) v2 from mds.? 10.252.24.88:6800/17213 -5658> 2013-04-03 10:47:17.397954 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21863 v187) v2 -- ?+0 0x2145580 con 0x200d420 -5657> 2013-04-03 10:47:17.398009 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114147 ==== paxos(lease_ack lc 427552 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (2007924302 0 0) 0x32df500 con 0x208a840 -5656> 2013-04-03 10:47:17.398056 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos -5655> 2013-04-03 10:47:17.398076 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394126 -5654> 2013-04-03 10:47:17.398093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5653> 2013-04-03 10:47:18.289874 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114148 ==== forward(pg_stats(1 pgs tid 48 v 605) v1) to leader v1 ==== 757+0+0 (2401160512 0 0) 0x273adc0 con 0x208a840 -5652> 2013-04-03 10:47:18.289901 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -5651> 2013-04-03 10:47:18.289912 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8000 from 10.252.24.88:6789/0 -5650> 2013-04-03 10:47:18.289964 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 preprocess_query pg_stats(1 pgs tid 48 v 605) v1 from osd.2 10.252.24.30:6801/5656 -5649> 2013-04-03 10:47:18.289988 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 prepare_update pg_stats(1 pgs tid 48 v 605) v1 from osd.2 10.252.24.30:6801/5656 -5648> 2013-04-03 10:47:18.289994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 prepare_pg_stats pg_stats(1 pgs tid 48 v 605) v1 from osd.2 -5647> 2013-04-03 10:47:18.290000 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29137 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -5646> 2013-04-03 10:47:18.340186 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29137 encode_pending v 29138 -5645> 2013-04-03 10:47:18.340222 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427552) queue_proposal bl 654 bytes; ctx = 0x2c3df80 -5644> 2013-04-03 10:47:18.340231 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427552) propose_queued 427553 654 bytes -5643> 2013-04-03 10:47:18.341902 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427552 fc 0 pn 17600 opn 0) v3 -- ?+0 0x32df280 -5642> 2013-04-03 10:47:18.377646 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114149 ==== paxos(accept lc 427552 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (318196415 0 0) 0x32df280 con 0x208a840 -5641> 2013-04-03 10:47:18.384152 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427553 fc 0 pn 17600 opn 0) v3 -- ?+0 0x32df500 -5640> 2013-04-03 10:47:18.384180 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427553 fc 427522 pn 0 opn 0) v3 -- ?+0 0x32dfa00 -5639> 2013-04-03 10:47:18.384252 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29137 update_from_paxos applying incremental 29138 -5638> 2013-04-03 10:47:18.384267 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 v29138: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2461B/s wr, 0op/s -5637> 2013-04-03 10:47:18.384285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 send_pg_creates to 0 pgs -5636> 2013-04-03 10:47:18.384287 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 update_logger -5635> 2013-04-03 10:47:18.384297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 create_pending v 29139 -5634> 2013-04-03 10:47:18.384309 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29138 _updated_stats for osd.2 10.252.24.30:6801/5656 -5633> 2013-04-03 10:47:18.384314 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 48) v1 tid 17781) v2 -- ?+0 0x2b26b40 con 0x208a840 -5632> 2013-04-03 10:47:18.384324 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 check_osd_map already seen 605 >= 605 -5631> 2013-04-03 10:47:18.384326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 update_logger -5630> 2013-04-03 10:47:18.384336 7fcdffcdb700 0 log [INF] : pgmap v29138: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2461B/s wr, 0op/s -5629> 2013-04-03 10:47:18.384338 7fcdffcdb700 10 send_log to self -5628> 2013-04-03 10:47:18.384339 7fcdffcdb700 10 log_queue is 1 last_log 204 sent 203 num 1 unsent 1 sending 1 -5627> 2013-04-03 10:47:18.384340 7fcdffcdb700 10 will send 2013-04-03 10:47:18.384337 mon.0 10.252.24.25:6789/0 204 : [INF] pgmap v29138: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2461B/s wr, 0op/s -5626> 2013-04-03 10:47:18.384347 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b74200 -5625> 2013-04-03 10:47:18.384393 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos -5624> 2013-04-03 10:47:18.384413 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394126 -5623> 2013-04-03 10:47:18.384431 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5622> 2013-04-03 10:47:18.384453 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b74200 con 0x1bac160 -5621> 2013-04-03 10:47:18.384473 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29138 check_down_pgs -5620> 2013-04-03 10:47:18.384491 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29138 v29138: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2461B/s wr, 0op/s -5619> 2013-04-03 10:47:18.384515 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -5618> 2013-04-03 10:47:18.384537 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -5617> 2013-04-03 10:47:18.384573 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -5616> 2013-04-03 10:47:18.384606 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos -5615> 2013-04-03 10:47:18.384622 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394126 -5614> 2013-04-03 10:47:18.384632 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394126 log -5613> 2013-04-03 10:47:18.384647 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5612> 2013-04-03 10:47:18.384664 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2061 auth -5611> 2013-04-03 10:47:18.384698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos -5610> 2013-04-03 10:47:18.384713 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394126 -5609> 2013-04-03 10:47:18.384722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -5608> 2013-04-03 10:47:18.384733 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 preprocess_log log(1 entries) v1 from mon.0 -5607> 2013-04-03 10:47:18.384756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -5606> 2013-04-03 10:47:18.384768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 prepare_log log(1 entries) v1 from mon.0 -5605> 2013-04-03 10:47:18.384785 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 logging 2013-04-03 10:47:18.384337 mon.0 10.252.24.25:6789/0 204 : [INF] pgmap v29138: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2461B/s wr, 0op/s -5604> 2013-04-03 10:47:18.399055 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114150 ==== paxos(lease_ack lc 427553 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3921590578 0 0) 0x32dfa00 con 0x208a840 -5603> 2013-04-03 10:47:18.399099 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos -5602> 2013-04-03 10:47:18.399116 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394126 update_from_paxos version 394126 summary v 394126 -5601> 2013-04-03 10:47:18.399131 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5600> 2013-04-03 10:47:19.377854 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394126 encode_pending v394127 -5599> 2013-04-03 10:47:19.377882 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427553) queue_proposal bl 443 bytes; ctx = 0x2c3dfc0 -5598> 2013-04-03 10:47:19.377890 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427553) propose_queued 427554 443 bytes -5597> 2013-04-03 10:47:19.379164 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427553 fc 0 pn 17600 opn 0) v3 -- ?+0 0x32df000 -5596> 2013-04-03 10:47:19.421519 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114151 ==== paxos(accept lc 427553 fc 0 pn 17600 opn 0) v3 ==== 80+0+0 (251565776 0 0) 0x32df000 con 0x208a840 -5595> 2013-04-03 10:47:19.421851 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427554 fc 0 pn 17600 opn 0) v3 -- ?+0 0x32dfa00 -5594> 2013-04-03 10:47:19.421878 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427554 fc 427522 pn 0 opn 0) v3 -- ?+0 0x32df280 -5593> 2013-04-03 10:47:19.421916 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos -5592> 2013-04-03 10:47:19.421945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394126 -5591> 2013-04-03 10:47:19.421959 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos latest full 393798 -5590> 2013-04-03 10:47:19.421976 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394127 update_from_paxos applying incremental log 394127 2013-04-03 10:47:18.384337 mon.0 10.252.24.25:6789/0 204 : [INF] pgmap v29138: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2461B/s wr, 0op/s -5589> 2013-04-03 10:47:19.422022 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 check_subs -5588> 2013-04-03 10:47:19.422052 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 create_pending v 394128 -5587> 2013-04-03 10:47:19.422074 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394127 _updated_log for mon.0 10.252.24.25:6789/0 -5586> 2013-04-03 10:47:19.422087 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 204) v1 -- ?+0 0x28a5380 con 0x1bac160 -5585> 2013-04-03 10:47:19.422139 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos -5584> 2013-04-03 10:47:19.422155 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 -5583> 2013-04-03 10:47:19.422174 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5582> 2013-04-03 10:47:19.422198 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 204) v1 ==== 0+0+0 (0 0 0) 0x28a5380 con 0x1bac160 -5581> 2013-04-03 10:47:19.422206 7fcdffcdb700 10 handle_log_ack log(last 204) v1 -5580> 2013-04-03 10:47:19.422208 7fcdffcdb700 10 logged 2013-04-03 10:47:18.384337 mon.0 10.252.24.25:6789/0 204 : [INF] pgmap v29138: 240 pgs: 240 active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 2461B/s wr, 0op/s -5579> 2013-04-03 10:47:19.434595 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 698114152 ==== paxos(lease_ack lc 427554 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1056259386 0 0) 0x32df280 con 0x208a840 -5578> 2013-04-03 10:47:19.434630 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos -5577> 2013-04-03 10:47:19.434646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 -5576> 2013-04-03 10:47:19.434661 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5575> 2013-04-03 10:47:19.600388 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 210 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140493 v187) v2 ==== 289+0+0 (4234694723 0 0) 0x20e4b80 con 0x2118420 -5574> 2013-04-03 10:47:19.600460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140493 v187) v2 from mds.? 10.252.58.31:6800/10908 -5573> 2013-04-03 10:47:19.600473 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140493 v187) v2 -- ?+0 0x273adc0 con 0x2118420 -5572> 2013-04-03 10:47:20.745413 7fce026be700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56643 s=2 pgs=151511 cs=1 l=0).reader couldn't read tag, Success -5571> 2013-04-03 10:47:20.745441 7fce026be700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56643 s=2 pgs=151511 cs=1 l=0).fault 0: Success -5570> 2013-04-03 10:47:20.745452 7fce026be700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56643 s=2 pgs=151511 cs=1 l=0).fault with nothing to send, going to standby -5569> 2013-04-03 10:47:20.745609 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x300c780 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33183/0 -5568> 2013-04-03 10:47:20.745665 7fcdfe6d3700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x303a780 sd=26 :6789 s=0 pgs=0 cs=0 l=0).accept sd=26 10.252.24.30:47278/0 -5567> 2013-04-03 10:47:20.745703 7fcdfe5d2700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x300c000 sd=27 :6789 s=0 pgs=0 cs=0 l=0).accept sd=27 10.252.24.25:41731/0 -5566> 2013-04-03 10:47:20.745743 7fcdfe5d2700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.25:0/3745741276 client protocol 0 -5565> 2013-04-03 10:47:20.745857 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11041 10.252.24.25:0/3745741276 1 ==== auth(proto 0 30 bytes epoch 5) v1 ==== 60+0+0 (637781556 0 0) 0x2523200 con 0x319b420 -5564> 2013-04-03 10:47:20.745871 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -5563> 2013-04-03 10:47:20.745876 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11041 10.252.24.25:0/3745741276 is open for client.11041 10.252.24.25:0/3745741276 -5562> 2013-04-03 10:47:20.745881 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -5561> 2013-04-03 10:47:20.745924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5560> 2013-04-03 10:47:20.745961 7fcdfe6d3700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.30:0/802047285 client protocol 0 -5559> 2013-04-03 10:47:20.746113 7fcdfe2cf700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3303c80 sd=29 :6789 s=0 pgs=0 cs=0 l=0).accept sd=29 10.252.24.32:39986/0 -5558> 2013-04-03 10:47:20.746396 7fcdfe2cf700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.32:0/3871608469 client protocol 0 -5557> 2013-04-03 10:47:20.745939 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11041 10.252.24.25:0/3745741276 -5556> 2013-04-03 10:47:20.746514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=30 -5555> 2013-04-03 10:47:20.746546 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:0/3745741276 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x2b74c00 con 0x319b420 -5554> 2013-04-03 10:47:20.746665 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11074 10.252.24.32:0/3871608469 1 ==== auth(proto 0 30 bytes epoch 5) v1 ==== 60+0+0 (593848846 0 0) 0x1c16e00 con 0x319b2c0 -5553> 2013-04-03 10:47:20.746676 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -5552> 2013-04-03 10:47:20.746680 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11074 10.252.24.32:0/3871608469 is open for client.11074 10.252.24.32:0/3871608469 -5551> 2013-04-03 10:47:20.746684 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -5550> 2013-04-03 10:47:20.746706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5549> 2013-04-03 10:47:20.746719 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11074 10.252.24.32:0/3871608469 -5548> 2013-04-03 10:47:20.746731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=30 -5547> 2013-04-03 10:47:20.746755 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.32:0/3871608469 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x2b75e00 con 0x319b2c0 -5546> 2013-04-03 10:47:20.746769 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11051 10.252.24.30:0/802047285 1 ==== auth(proto 0 30 bytes epoch 5) v1 ==== 60+0+0 (3397564709 0 0) 0x1c16a00 con 0x319b000 -5545> 2013-04-03 10:47:20.746776 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -5544> 2013-04-03 10:47:20.746778 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11051 10.252.24.30:0/802047285 is open for client.11051 10.252.24.30:0/802047285 -5543> 2013-04-03 10:47:20.746782 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -5542> 2013-04-03 10:47:20.746800 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5541> 2013-04-03 10:47:20.746814 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11051 10.252.24.30:0/802047285 -5540> 2013-04-03 10:47:20.746823 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=30 -5539> 2013-04-03 10:47:20.746843 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:0/802047285 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x2b74400 con 0x319b000 -5538> 2013-04-03 10:47:20.746855 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11041 10.252.24.25:0/3745741276 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (2015570526 0 0) 0x209c200 con 0x319b420 -5537> 2013-04-03 10:47:20.746879 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5536> 2013-04-03 10:47:20.746895 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.11041 10.252.24.25:0/3745741276 -5535> 2013-04-03 10:47:20.746906 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=128 -5534> 2013-04-03 10:47:20.747046 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:0/3745741276 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x1c16a00 con 0x319b420 -5533> 2013-04-03 10:47:20.747256 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11041 10.252.24.25:0/3745741276 3 ==== mon_subscribe({mdsmap=188+,monmap=6+,osdmap=606}) v2 ==== 61+0+0 (768048925 0 0) 0x3306000 con 0x319b420 -5532> 2013-04-03 10:47:20.747270 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+,osdmap=606}) v2 -5531> 2013-04-03 10:47:20.747279 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -5530> 2013-04-03 10:47:20.747284 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11041 10.252.24.25:0/3745741276 -- mon_subscribe_ack(300s) v1 -- ?+0 0x28a5380 -5529> 2013-04-03 10:47:20.747369 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11051 10.252.24.30:0/802047285 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (1536754072 0 0) 0x1c16a00 con 0x319b000 -5528> 2013-04-03 10:47:20.747407 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5527> 2013-04-03 10:47:20.747424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.11051 10.252.24.30:0/802047285 -5526> 2013-04-03 10:47:20.747435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=128 -5525> 2013-04-03 10:47:20.747559 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:0/802047285 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x209c200 con 0x319b000 -5524> 2013-04-03 10:47:20.747576 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11074 10.252.24.32:0/3871608469 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (3350423454 0 0) 0x209c600 con 0x319b2c0 -5523> 2013-04-03 10:47:20.747603 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5522> 2013-04-03 10:47:20.747619 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.11074 10.252.24.32:0/3871608469 -5521> 2013-04-03 10:47:20.747631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=128 -5520> 2013-04-03 10:47:20.747741 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.32:0/3871608469 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x1c16a00 con 0x319b2c0 -5519> 2013-04-03 10:47:20.748081 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11051 10.252.24.30:0/802047285 3 ==== mon_subscribe({mdsmap=188+,monmap=6+}) v2 ==== 42+0+0 (435898880 0 0) 0x33061c0 con 0x319b000 -5518> 2013-04-03 10:47:20.748101 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 -5517> 2013-04-03 10:47:20.748116 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -5516> 2013-04-03 10:47:20.748119 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11051 10.252.24.30:0/802047285 -- mon_subscribe_ack(300s) v1 -- ?+0 0x3306000 -5515> 2013-04-03 10:47:20.748428 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11074 10.252.24.32:0/3871608469 3 ==== mon_subscribe({mdsmap=188+,monmap=6+}) v2 ==== 42+0+0 (435898880 0 0) 0x3306380 con 0x319b2c0 -5514> 2013-04-03 10:47:20.748445 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 -5513> 2013-04-03 10:47:20.748457 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -5512> 2013-04-03 10:47:20.748461 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11074 10.252.24.32:0/3871608469 -- mon_subscribe_ack(300s) v1 -- ?+0 0x33061c0 -5511> 2013-04-03 10:47:20.748988 7fcdfe7d4700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.88:6801/17538 osd protocol 0 -5510> 2013-04-03 10:47:20.749286 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6801/17538 1 ==== auth(proto 0 26 bytes epoch 5) v1 ==== 56+0+0 (2056785542 0 0) 0x1c16a00 con 0x319b160 -5509> 2013-04-03 10:47:20.749305 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -5508> 2013-04-03 10:47:20.749314 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: osd.3 10.252.24.88:6801/17538 is open for osd.3 10.252.24.88:6801/17538 -5507> 2013-04-03 10:47:20.749323 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -5506> 2013-04-03 10:47:20.749345 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5505> 2013-04-03 10:47:20.749359 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 0 26 bytes epoch 5) v1 from osd.3 10.252.24.88:6801/17538 -5504> 2013-04-03 10:47:20.749371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=26 -5503> 2013-04-03 10:47:20.749393 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6801/17538 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x2b74800 con 0x319b160 -5502> 2013-04-03 10:47:20.749894 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6801/17538 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (3729610323 0 0) 0x209c200 con 0x319b160 -5501> 2013-04-03 10:47:20.749921 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5500> 2013-04-03 10:47:20.749937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from osd.3 10.252.24.88:6801/17538 -5499> 2013-04-03 10:47:20.749949 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=128 -5498> 2013-04-03 10:47:20.750062 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6801/17538 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x1c16a00 con 0x319b160 -5497> 2013-04-03 10:47:20.750517 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6801/17538 3 ==== mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=603}) v2 ==== 69+0+0 (976368017 0 0) 0x3306540 con 0x319b160 -5496> 2013-04-03 10:47:20.750531 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=603}) v2 -5495> 2013-04-03 10:47:20.750537 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -5494> 2013-04-03 10:47:20.750546 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e605 send_incremental [603..605] to osd.3 10.252.24.88:6801/17538 -5493> 2013-04-03 10:47:20.750570 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 build_incremental [603..605] -5492> 2013-04-03 10:47:20.750606 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.3 10.252.24.88:6801/17538 -- osd_map(603..605 src has 104..605) v3 -- ?+0 0x209c200 -5491> 2013-04-03 10:47:20.750619 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.3 10.252.24.88:6801/17538 -- mon_subscribe_ack(300s) v1 -- ?+0 0x3306380 -5490> 2013-04-03 10:47:20.750677 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6801/17538 4 ==== pg_stats(1 pgs tid 30464 v 605) v1 ==== 485+0+0 (599038730 0 0) 0x330a000 con 0x319b160 -5489> 2013-04-03 10:47:20.750722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 preprocess_query pg_stats(1 pgs tid 30464 v 605) v1 from osd.3 10.252.24.88:6801/17538 -5488> 2013-04-03 10:47:20.750747 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 prepare_update pg_stats(1 pgs tid 30464 v 605) v1 from osd.3 10.252.24.88:6801/17538 -5487> 2013-04-03 10:47:20.750753 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 prepare_pg_stats pg_stats(1 pgs tid 30464 v 605) v1 from osd.3 -5486> 2013-04-03 10:47:20.750759 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29138 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -5485> 2013-04-03 10:47:20.750785 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6801/17538 5 ==== mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=603}) v2 ==== 69+0+0 (976368017 0 0) 0x33068c0 con 0x319b160 -5484> 2013-04-03 10:47:20.750795 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=603}) v2 -5483> 2013-04-03 10:47:20.750800 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -5482> 2013-04-03 10:47:20.750804 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e605 send_incremental [603..605] to osd.3 10.252.24.88:6801/17538 -5481> 2013-04-03 10:47:20.750818 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 build_incremental [603..605] -5480> 2013-04-03 10:47:20.750852 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.3 10.252.24.88:6801/17538 -- osd_map(603..605 src has 104..605) v3 -- ?+0 0x2b74e00 -5479> 2013-04-03 10:47:20.750862 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.3 10.252.24.88:6801/17538 -- mon_subscribe_ack(300s) v1 -- ?+0 0x28a51c0 -5478> 2013-04-03 10:47:20.801508 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29138 encode_pending v 29139 -5477> 2013-04-03 10:47:20.801544 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427554) queue_proposal bl 654 bytes; ctx = 0x2c3de20 -5476> 2013-04-03 10:47:20.801553 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427554) propose_queued 427555 654 bytes -5475> 2013-04-03 10:47:20.803901 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427554 fc 0 pn 17600 opn 0) v3 -- ?+0 0x32df000 -5474> 2013-04-03 10:47:20.804183 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56643 s=1 pgs=151511 cs=2 l=0).connect error 10.252.24.88:6789/0, 111: Connection refused -5473> 2013-04-03 10:47:20.804202 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56643 s=1 pgs=151511 cs=2 l=0).fault 111: Connection refused -5472> 2013-04-03 10:47:20.804210 7fce027bf700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56643 s=1 pgs=151511 cs=2 l=0).fault -5471> 2013-04-03 10:47:20.804385 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56643 s=1 pgs=151511 cs=2 l=0).connect error 10.252.24.88:6789/0, 111: Connection refused -5470> 2013-04-03 10:47:20.804409 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56643 s=1 pgs=151511 cs=2 l=0).fault 111: Connection refused -5469> 2013-04-03 10:47:21.004668 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56643 s=1 pgs=151511 cs=2 l=0).connect error 10.252.24.88:6789/0, 111: Connection refused -5468> 2013-04-03 10:47:21.004697 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56643 s=1 pgs=151511 cs=2 l=0).fault 111: Connection refused -5467> 2013-04-03 10:47:21.398009 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/17213 210 ==== mdsbeacon(11006/hotcpc12533 up:standby seq 21864 v187) v2 ==== 289+0+0 (930520752 0 0) 0x2145000 con 0x200d420 -5466> 2013-04-03 10:47:21.398086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11006/hotcpc12533 up:standby seq 21864 v187) v2 from mds.? 10.252.24.88:6800/17213 -5465> 2013-04-03 10:47:21.398099 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/17213 -- mdsbeacon(11006/hotcpc12533 up:standby seq 21864 v187) v2 -- ?+0 0x20e4b80 con 0x200d420 -5464> 2013-04-03 10:47:21.404952 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56643 s=1 pgs=151511 cs=2 l=0).connect error 10.252.24.88:6789/0, 111: Connection refused -5463> 2013-04-03 10:47:21.404984 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56643 s=1 pgs=151511 cs=2 l=0).fault 111: Connection refused -5462> 2013-04-03 10:47:21.808082 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6801/17538 6 ==== pg_stats(1 pgs tid 30465 v 605) v1 ==== 485+0+0 (4245903398 0 0) 0x20ad900 con 0x319b160 -5461> 2013-04-03 10:47:22.225006 7fce027bf700 10 mon.hotcpc9039@0(leader) e5 ms_get_authorizer for mon -5460> 2013-04-03 10:47:22.225283 7fce026be700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3303a00 sd=28 :6789 s=0 pgs=0 cs=0 l=0).accept sd=28 10.252.24.88:33185/0 -5459> 2013-04-03 10:47:22.225798 7fce026be700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.88:6789/0 mon protocol 2 -5458> 2013-04-03 10:47:22.226206 7fce026be700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.88:6789/0 mon protocol 2 -5457> 2013-04-03 10:47:22.226304 7fce026be700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=0 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 2 state connecting -5456> 2013-04-03 10:47:22.226314 7fce026be700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=0 pgs=0 cs=0 l=0).accept peer reset, then tried to connect to us, replacing -5455> 2013-04-03 10:47:22.226346 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56786 s=4 pgs=151511 cs=0 l=0).connect read reply Transport endpoint is not connected -5454> 2013-04-03 10:47:22.226364 7fce027bf700 3 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x1f91a00 sd=21 :56786 s=4 pgs=151511 cs=0 l=0).connect fault, but state = closed != connecting, stopping -5453> 2013-04-03 10:47:22.226744 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 1 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc12533) v4 ==== 96+0+0 (650395888 0 0) 0x32ee000 con 0x208a840 -5452> 2013-04-03 10:47:22.226760 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc12533) v4 -5451> 2013-04-03 10:47:22.226770 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_probe_probe mon.2 10.252.24.88:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc12533) v4 -5450> 2013-04-03 10:47:22.226780 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,2 paxos( fc 427522 lc 427554 )) v4 -- ?+0 0x32eec00 con 0x208a840 -5449> 2013-04-03 10:47:22.265969 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 2 ==== election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 propose 353) v3 ==== 537+0+0 (4122014838 0 0) 0x20add80 con 0x208a840 -5448> 2013-04-03 10:47:22.266007 7fcdffcdb700 5 mon.hotcpc9039@0(leader).elector(352) handle_propose from mon.2 -5447> 2013-04-03 10:47:22.266012 7fcdffcdb700 10 mon.hotcpc9039@0(leader).elector(352) bump_epoch 352 to 353 -5446> 2013-04-03 10:47:22.267655 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 reset -5445> 2013-04-03 10:47:22.267671 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish -5444> 2013-04-03 10:47:22.267726 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 start_election -5443> 2013-04-03 10:47:22.267730 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 cancel_probe_timeout (none scheduled) -5442> 2013-04-03 10:47:22.267736 7fcdffcdb700 0 log [INF] : mon.hotcpc9039 calling new monitor election -5441> 2013-04-03 10:47:22.267738 7fcdffcdb700 10 send_log to self -5440> 2013-04-03 10:47:22.267739 7fcdffcdb700 10 log_queue is 1 last_log 205 sent 204 num 1 unsent 1 sending 1 -5439> 2013-04-03 10:47:22.267740 7fcdffcdb700 10 will send 2013-04-03 10:47:22.267737 mon.0 10.252.24.25:6789/0 205 : [INF] mon.hotcpc9039 calling new monitor election -5438> 2013-04-03 10:47:22.267749 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b75400 -5437> 2013-04-03 10:47:22.267756 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(353) start -- can i be leader? -5436> 2013-04-03 10:47:22.267770 7fcdffcdb700 1 mon.hotcpc9039@0(electing).elector(353) init, last seen epoch 353 -5435> 2013-04-03 10:47:22.267778 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 propose 353) v3 -- ?+0 0x24f8000 -5434> 2013-04-03 10:47:22.267788 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 propose 353) v3 -- ?+0 0x24f8d80 -5433> 2013-04-03 10:47:22.267802 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75400 con 0x1bac160 -5432> 2013-04-03 10:47:22.267835 7fcdffcdb700 1 mon.hotcpc9039@0(electing).paxos(paxos recovering c 427522..427554) is_readable now=2013-04-03 10:47:22.267835 lease_expire=2013-04-03 10:47:24.421876 has v0 lc 427554 -5431> 2013-04-03 10:47:22.268603 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 3 ==== election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 ack 353) v3 ==== 537+0+0 (327180824 0 0) 0x20adb40 con 0x208a840 -5430> 2013-04-03 10:47:22.268621 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(353) handle_ack from mon.2 -5429> 2013-04-03 10:47:22.268624 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(353) so far i have {0=4294967295,2=4294967295} -5428> 2013-04-03 10:47:22.296583 7fcdfead7700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6800/17213 pipe(0x2116c80 sd=23 :6789 s=2 pgs=29912 cs=1 l=1).reader couldn't read tag, Success -5427> 2013-04-03 10:47:22.296616 7fcdfead7700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6800/17213 pipe(0x2116c80 sd=23 :6789 s=2 pgs=29912 cs=1 l=1).fault 0: Success -5426> 2013-04-03 10:47:22.296662 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 ms_handle_reset 0x200d420 10.252.24.88:6800/17213 -5425> 2013-04-03 10:47:22.296671 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 reset/close on session mds.? 10.252.24.88:6800/17213 -5424> 2013-04-03 10:47:22.296678 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 remove_session MonSession: mds.? 10.252.24.88:6800/17213 is openallow rwx mds.? 10.252.24.88:6800/17213 -5423> 2013-04-03 10:47:22.964067 7fcdfead7700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3303780 sd=21 :6789 s=0 pgs=0 cs=0 l=0).accept sd=21 10.252.24.30:47284/0 -5422> 2013-04-03 10:47:22.964348 7fcdfead7700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.24.30:6801/5656 osd protocol 0 -5421> 2013-04-03 10:47:22.964723 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.2 10.252.24.30:6801/5656 1 ==== auth(proto 0 26 bytes epoch 5) v1 ==== 56+0+0 (1815740086 0 0) 0x25f0200 con 0x3311c60 -5420> 2013-04-03 10:47:22.964742 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 26 bytes epoch 5) v1 until we get in quorum -5419> 2013-04-03 10:47:23.384816 7fce004dc700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 26 bytes epoch 5) v1 until we get in quorum -5418> 2013-04-03 10:47:23.431301 7fcdfdfcc700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3303500 sd=23 :6789 s=0 pgs=0 cs=0 l=0).accept sd=23 10.252.24.88:33187/0 -5417> 2013-04-03 10:47:23.431614 7fcdfdfcc700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.24.88:6800/18040 mds protocol 0 -5416> 2013-04-03 10:47:23.431940 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 1 ==== auth(proto 0 36 bytes epoch 0) v1 ==== 66+0+0 (2489530022 0 0) 0x2201200 con 0x3311b00 -5415> 2013-04-03 10:47:23.431968 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 36 bytes epoch 0) v1 until we get in quorum -5414> 2013-04-03 10:47:23.600513 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 211 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140494 v187) v2 ==== 289+0+0 (2581303748 0 0) 0x20e48c0 con 0x2118420 -5413> 2013-04-03 10:47:23.600574 7fcdffcdb700 1 mon.hotcpc9039@0(electing).paxos(paxos recovering c 427522..427554) is_readable now=2013-04-03 10:47:23.600575 lease_expire=2013-04-03 10:47:24.421876 has v0 lc 427554 -5412> 2013-04-03 10:47:24.061835 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6801/17538 pipe(0x300c780 sd=25 :6789 s=2 pgs=21695 cs=1 l=1).reader couldn't read tag, Success -5411> 2013-04-03 10:47:24.061860 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6801/17538 pipe(0x300c780 sd=25 :6789 s=2 pgs=21695 cs=1 l=1).fault 0: Success -5410> 2013-04-03 10:47:24.061895 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 ms_handle_reset 0x319b160 10.252.24.88:6801/17538 -5409> 2013-04-03 10:47:24.061909 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 reset/close on session osd.3 10.252.24.88:6801/17538 -5408> 2013-04-03 10:47:24.061913 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 remove_session MonSession: osd.3 10.252.24.88:6801/17538 is openallow rwx osd.3 10.252.24.88:6801/17538 -5407> 2013-04-03 10:47:24.062060 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11074 10.252.24.32:0/3871608469 4 ==== mon_subscribe({mdsmap=188+,monmap=6+,osdmap=606}) v2 ==== 61+0+0 (768048925 0 0) 0x2e8b000 con 0x319b2c0 -5406> 2013-04-03 10:47:24.062075 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+,osdmap=606}) v2 -5405> 2013-04-03 10:47:24.062082 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 check_sub monmap next 6 have 5 -5404> 2013-04-03 10:47:24.062087 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11074 10.252.24.32:0/3871608469 -- mon_subscribe_ack(300s) v1 -- ?+0 0x3306540 -5403> 2013-04-03 10:47:24.449116 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3303280 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.30:47298/0 -5402> 2013-04-03 10:47:24.449410 7fcdfe7d4700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.24.30:0/517 client protocol 0 -5401> 2013-04-03 10:47:24.449719 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11802 10.252.24.30:0/517 1 ==== auth(proto 0 30 bytes epoch 5) v1 ==== 60+0+0 (300400647 0 0) 0x2b74e00 con 0x33119a0 -5400> 2013-04-03 10:47:24.449749 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 30 bytes epoch 5) v1 until we get in quorum -5399> 2013-04-03 10:47:24.538023 7fcdfddca700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3303000 sd=30 :6789 s=0 pgs=0 cs=0 l=0).accept sd=30 10.252.24.25:41754/0 -5398> 2013-04-03 10:47:24.538055 7fcdfdcc9700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2116c80 sd=31 :6789 s=0 pgs=0 cs=0 l=0).accept sd=31 10.252.24.25:41755/0 -5397> 2013-04-03 10:47:24.538078 7fcdfddca700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.24.25:0/2614196857 client protocol 0 -5396> 2013-04-03 10:47:24.538085 7fcdfdcc9700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.24.25:0/2151250553 client protocol 0 -5395> 2013-04-03 10:47:24.538207 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11042 10.252.24.25:0/2614196857 1 ==== auth(proto 0 30 bytes epoch 5) v1 ==== 60+0+0 (1972484708 0 0) 0x2b5b200 con 0x3311840 -5394> 2013-04-03 10:47:24.538221 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 30 bytes epoch 5) v1 until we get in quorum -5393> 2013-04-03 10:47:24.538227 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11045 10.252.24.25:0/2151250553 1 ==== auth(proto 0 30 bytes epoch 5) v1 ==== 60+0+0 (3821311732 0 0) 0x2b78600 con 0x33116e0 -5392> 2013-04-03 10:47:24.538232 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 30 bytes epoch 5) v1 until we get in quorum -5391> 2013-04-03 10:47:24.694732 7fcdfd9c6700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3062280 sd=32 :6789 s=0 pgs=0 cs=0 l=0).accept sd=32 10.252.58.31:44180/0 -5390> 2013-04-03 10:47:24.695111 7fcdfd9c6700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.58.31:6802/2682 osd protocol 0 -5389> 2013-04-03 10:47:24.695436 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.0 10.252.58.31:6802/2682 1 ==== auth(proto 0 26 bytes epoch 5) v1 ==== 56+0+0 (3150273078 0 0) 0x2b78000 con 0x3311580 -5388> 2013-04-03 10:47:24.695464 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 26 bytes epoch 5) v1 until we get in quorum -5387> 2013-04-03 10:47:24.860255 7fcdfd7c4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3062780 sd=33 :6789 s=0 pgs=0 cs=0 l=0).accept sd=33 10.252.24.32:39991/0 -5386> 2013-04-03 10:47:24.860538 7fcdfd7c4700 10 mon.hotcpc9039@0(electing) e5 ms_verify_authorizer 10.252.24.32:0/1396772501 client protocol 0 -5385> 2013-04-03 10:47:24.860829 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11071 10.252.24.32:0/1396772501 1 ==== auth(proto 0 30 bytes epoch 5) v1 ==== 60+0+0 (376189430 0 0) 0x2b78400 con 0x3311420 -5384> 2013-04-03 10:47:24.860851 7fcdffcdb700 5 mon.hotcpc9039@0(electing) e5 waitlisting message auth(proto 0 30 bytes epoch 5) v1 until we get in quorum -5383> 2013-04-03 10:47:25.963953 7fcdfead7700 2 -- 10.252.24.25:6789/0 >> 10.252.24.30:6801/5656 pipe(0x3303780 sd=21 :6789 s=2 pgs=11 cs=1 l=1).reader couldn't read tag, Success -5382> 2013-04-03 10:47:25.963992 7fcdfead7700 2 -- 10.252.24.25:6789/0 >> 10.252.24.30:6801/5656 pipe(0x3303780 sd=21 :6789 s=2 pgs=11 cs=1 l=1).fault 0: Success -5381> 2013-04-03 10:47:25.964036 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 ms_handle_reset 0x3311c60 10.252.24.30:6801/5656 -5380> 2013-04-03 10:47:26.431094 7fcdfdfcc700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6800/18040 pipe(0x3303500 sd=23 :6789 s=2 pgs=1 cs=1 l=1).reader couldn't read tag, Success -5379> 2013-04-03 10:47:26.431142 7fcdfdfcc700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6800/18040 pipe(0x3303500 sd=23 :6789 s=2 pgs=1 cs=1 l=1).fault 0: Success -5378> 2013-04-03 10:47:26.431177 7fcdffcdb700 10 mon.hotcpc9039@0(electing) e5 ms_handle_reset 0x3311b00 10.252.24.88:6800/18040 -5377> 2013-04-03 10:47:27.267869 7fce004dc700 5 mon.hotcpc9039@0(electing).elector(353) election timer expired -5376> 2013-04-03 10:47:27.267890 7fce004dc700 10 mon.hotcpc9039@0(electing).elector(353) bump_epoch 353 to 354 -5375> 2013-04-03 10:47:27.270591 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 victory 354) v3 -- ?+0 0x24f86c0 -5374> 2013-04-03 10:47:27.270622 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 win_election, epoch 354 quorum is 0,2 features are 4294967295 -5373> 2013-04-03 10:47:27.270632 7fce004dc700 0 log [INF] : mon.hotcpc9039@0 won leader election with quorum 0,2 -5372> 2013-04-03 10:47:27.270636 7fce004dc700 10 send_log to self -5371> 2013-04-03 10:47:27.270637 7fce004dc700 10 log_queue is 2 last_log 206 sent 205 num 2 unsent 1 sending 1 -5370> 2013-04-03 10:47:27.270639 7fce004dc700 10 will send 2013-04-03 10:47:22.267737 mon.0 10.252.24.25:6789/0 205 : [INF] mon.hotcpc9039 calling new monitor election -5369> 2013-04-03 10:47:27.270646 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2b5b000 -5368> 2013-04-03 10:47:27.270706 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2b5b000 con 0x1bac160 -5367> 2013-04-03 10:47:27.270953 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(collect lc 427554 fc 427522 pn 17700 opn 0) v3 -- ?+0 0x32df280 -5366> 2013-04-03 10:47:27.271026 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427522..427554) is_readable now=2013-04-03 10:47:27.271028 lease_expire=0.000000 has v0 lc 427554 -5365> 2013-04-03 10:47:27.271056 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427522..427554) is_readable now=2013-04-03 10:47:27.271056 lease_expire=0.000000 has v0 lc 427554 -5364> 2013-04-03 10:47:27.271064 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish -5363> 2013-04-03 10:47:27.271069 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -5362> 2013-04-03 10:47:27.271072 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: osd.2 10.252.24.30:6801/5656 is open for osd.2 10.252.24.30:6801/5656 -5361> 2013-04-03 10:47:27.271077 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -5360> 2013-04-03 10:47:27.271102 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427522..427554) is_readable now=2013-04-03 10:47:27.271103 lease_expire=0.000000 has v0 lc 427554 -5359> 2013-04-03 10:47:27.271108 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -5358> 2013-04-03 10:47:27.271110 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: mds.? 10.252.24.88:6800/18040 is open for mds.? 10.252.24.88:6800/18040 -5357> 2013-04-03 10:47:27.271115 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -5356> 2013-04-03 10:47:27.271135 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427522..427554) is_readable now=2013-04-03 10:47:27.271135 lease_expire=0.000000 has v0 lc 427554 -5355> 2013-04-03 10:47:27.271140 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -5354> 2013-04-03 10:47:27.271144 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11802 10.252.24.30:0/517 is open for client.11802 10.252.24.30:0/517 -5353> 2013-04-03 10:47:27.271148 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -5352> 2013-04-03 10:47:27.271168 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427522..427554) is_readable now=2013-04-03 10:47:27.271169 lease_expire=0.000000 has v0 lc 427554 -5351> 2013-04-03 10:47:27.271174 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -5350> 2013-04-03 10:47:27.271177 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11042 10.252.24.25:0/2614196857 is open for client.11042 10.252.24.25:0/2614196857 -5349> 2013-04-03 10:47:27.271181 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -5348> 2013-04-03 10:47:27.271201 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427522..427554) is_readable now=2013-04-03 10:47:27.271202 lease_expire=0.000000 has v0 lc 427554 -5347> 2013-04-03 10:47:27.271208 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -5346> 2013-04-03 10:47:27.271211 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11045 10.252.24.25:0/2151250553 is open for client.11045 10.252.24.25:0/2151250553 -5345> 2013-04-03 10:47:27.271216 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -5344> 2013-04-03 10:47:27.271236 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427522..427554) is_readable now=2013-04-03 10:47:27.271237 lease_expire=0.000000 has v0 lc 427554 -5343> 2013-04-03 10:47:27.271244 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -5342> 2013-04-03 10:47:27.271247 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: osd.0 10.252.58.31:6802/2682 is open for osd.0 10.252.58.31:6802/2682 -5341> 2013-04-03 10:47:27.271251 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -5340> 2013-04-03 10:47:27.271271 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427522..427554) is_readable now=2013-04-03 10:47:27.271272 lease_expire=0.000000 has v0 lc 427554 -5339> 2013-04-03 10:47:27.271277 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -5338> 2013-04-03 10:47:27.271279 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11071 10.252.24.32:0/1396772501 is open for client.11071 10.252.24.32:0/1396772501 -5337> 2013-04-03 10:47:27.271283 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -5336> 2013-04-03 10:47:27.271304 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427522..427554) is_readable now=2013-04-03 10:47:27.271306 lease_expire=0.000000 has v0 lc 427554 -5335> 2013-04-03 10:47:27.271311 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 resend_routed_requests -5334> 2013-04-03 10:47:27.271313 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 register_cluster_logger - already registered -5333> 2013-04-03 10:47:27.271316 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start -5332> 2013-04-03 10:47:27.271317 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round curr 0 -5331> 2013-04-03 10:47:27.271318 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round new 1 -5330> 2013-04-03 10:47:27.271320 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck -5329> 2013-04-03 10:47:27.271321 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck start timecheck epoch 354 round 1 -5328> 2013-04-03 10:47:27.271325 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck send time_check( ping e 354 r 1 ) v1 to mon.2 10.252.24.88:6789/0 -5327> 2013-04-03 10:47:27.271330 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- time_check( ping e 354 r 1 ) v1 -- ?+0 0x24f8480 -5326> 2013-04-03 10:47:27.271336 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round setting up next event -5325> 2013-04-03 10:47:27.271407 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427522..427554) is_readable now=2013-04-03 10:47:27.271409 lease_expire=0.000000 has v0 lc 427554 -5324> 2013-04-03 10:47:27.449036 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.30:0/517 pipe(0x3303280 sd=25 :6789 s=2 pgs=11 cs=1 l=1).reader couldn't read tag, Success -5323> 2013-04-03 10:47:27.449083 7fcdfe7d4700 2 -- 10.252.24.25:6789/0 >> 10.252.24.30:0/517 pipe(0x3303280 sd=25 :6789 s=2 pgs=11 cs=1 l=1).fault 0: Success -5322> 2013-04-03 10:47:27.449126 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_handle_reset 0x33119a0 10.252.24.30:0/517 -5321> 2013-04-03 10:47:27.449139 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 reset/close on session client.11802 10.252.24.30:0/517 -5320> 2013-04-03 10:47:27.449145 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 remove_session MonSession: client.11802 10.252.24.30:0/517 is open client.11802 10.252.24.30:0/517 -5319> 2013-04-03 10:47:27.463995 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 4 ==== paxos(last lc 427554 fc 427522 pn 17700 opn 0) v3 ==== 80+0+0 (2352726383 0 0) 0x3303500 con 0x208a840 -5318> 2013-04-03 10:47:27.464359 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427554 fc 0 pn 17700 opn 0) v3 -- ?+0 0x32df780 -5317> 2013-04-03 10:47:27.464396 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 5 ==== time_check( pong e 354 r 1 ts 2013-04-03 10:47:27.462969 ) v1 ==== 36+0+0 (3854155039 0 0) 0x20fcb40 con 0x208a840 -5316> 2013-04-03 10:47:27.464408 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck time_check( pong e 354 r 1 ts 2013-04-03 10:47:27.462969 ) v1 -5315> 2013-04-03 10:47:27.464413 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader time_check( pong e 354 r 1 ts 2013-04-03 10:47:27.462969 ) v1 -5314> 2013-04-03 10:47:27.464420 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader from mon.2 10.252.24.88:6789/0 ts 2013-04-03 10:47:27.462969 delta -0.00144696 skew_bound 0 latency 0.193092 -5313> 2013-04-03 10:47:27.464432 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader got pongs from everybody (2 total) -5312> 2013-04-03 10:47:27.464434 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish_round curr 1 -5311> 2013-04-03 10:47:27.464436 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_report -5310> 2013-04-03 10:47:27.464440 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_report send report to mon.2 10.252.24.88:6789/0 -5309> 2013-04-03 10:47:27.464443 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- time_check( report e 354 r 2 #skews 2 #latencies 2 ) v1 -- ?+0 0x20adb40 -5308> 2013-04-03 10:47:27.477092 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 6 ==== paxos(accept lc 427554 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (181738356 0 0) 0x32df780 con 0x208a840 -5307> 2013-04-03 10:47:27.482664 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427555 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3303500 -5306> 2013-04-03 10:47:27.482692 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427555 fc 427522 pn 0 opn 0) v3 -- ?+0 0x300cc80 -5305> 2013-04-03 10:47:27.482783 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29138 update_from_paxos applying incremental 29139 -5304> 2013-04-03 10:47:27.482802 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 v29139: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 686B/s wr, 0op/s -5303> 2013-04-03 10:47:27.482821 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 send_pg_creates to 0 pgs -5302> 2013-04-03 10:47:27.482825 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 update_logger -5301> 2013-04-03 10:47:27.482835 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 create_pending v 29140 -5300> 2013-04-03 10:47:27.483202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 check_osd_map already seen 605 >= 605 -5299> 2013-04-03 10:47:27.483205 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 update_logger -5298> 2013-04-03 10:47:27.483217 7fcdffcdb700 0 log [INF] : pgmap v29139: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 686B/s wr, 0op/s -5297> 2013-04-03 10:47:27.483219 7fcdffcdb700 10 send_log to self -5296> 2013-04-03 10:47:27.483220 7fcdffcdb700 10 log_queue is 3 last_log 207 sent 205 num 3 unsent 2 sending 2 -5295> 2013-04-03 10:47:27.483222 7fcdffcdb700 10 will send 2013-04-03 10:47:22.267737 mon.0 10.252.24.25:6789/0 205 : [INF] mon.hotcpc9039 calling new monitor election -5294> 2013-04-03 10:47:27.483228 7fcdffcdb700 10 will send 2013-04-03 10:47:27.270635 mon.0 10.252.24.25:6789/0 206 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 -5293> 2013-04-03 10:47:27.483232 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b75600 -5292> 2013-04-03 10:47:27.483266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 create_pending e188 -5291> 2013-04-03 10:47:27.483285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -5290> 2013-04-03 10:47:27.483296 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 update_logger -5289> 2013-04-03 10:47:27.483301 7fcdffcdb700 0 log [INF] : mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -5288> 2013-04-03 10:47:27.483302 7fcdffcdb700 10 send_log to self -5287> 2013-04-03 10:47:27.483303 7fcdffcdb700 10 log_queue is 4 last_log 208 sent 206 num 4 unsent 2 sending 2 -5286> 2013-04-03 10:47:27.483304 7fcdffcdb700 10 will send 2013-04-03 10:47:27.270635 mon.0 10.252.24.25:6789/0 206 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 -5285> 2013-04-03 10:47:27.483309 7fcdffcdb700 10 will send 2013-04-03 10:47:27.483218 mon.0 10.252.24.25:6789/0 207 : [INF] pgmap v29139: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 686B/s wr, 0op/s -5284> 2013-04-03 10:47:27.483312 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2201000 -5283> 2013-04-03 10:47:27.483340 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 create_pending e 606 -5282> 2013-04-03 10:47:27.483349 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 remove_redundant_pg_temp -5281> 2013-04-03 10:47:27.483351 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 remove_down_pg_temp -5280> 2013-04-03 10:47:27.483371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 update_logger -5279> 2013-04-03 10:47:27.483377 7fcdffcdb700 0 log [INF] : osdmap e605: 5 osds: 5 up, 5 in -5278> 2013-04-03 10:47:27.483378 7fcdffcdb700 10 send_log to self -5277> 2013-04-03 10:47:27.483378 7fcdffcdb700 10 log_queue is 5 last_log 209 sent 207 num 5 unsent 2 sending 2 -5276> 2013-04-03 10:47:27.483380 7fcdffcdb700 10 will send 2013-04-03 10:47:27.483218 mon.0 10.252.24.25:6789/0 207 : [INF] pgmap v29139: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 686B/s wr, 0op/s -5275> 2013-04-03 10:47:27.483385 7fcdffcdb700 10 will send 2013-04-03 10:47:27.483302 mon.0 10.252.24.25:6789/0 208 : [INF] mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -5274> 2013-04-03 10:47:27.483388 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x209c600 -5273> 2013-04-03 10:47:27.483393 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos -5272> 2013-04-03 10:47:27.483412 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 -5271> 2013-04-03 10:47:27.483437 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 create_pending v 394128 -5270> 2013-04-03 10:47:27.483475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).monmap v5 create_pending monmap epoch 6 -5269> 2013-04-03 10:47:27.483498 7fcdffcdb700 0 log [INF] : monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -5268> 2013-04-03 10:47:27.483501 7fcdffcdb700 10 send_log to self -5267> 2013-04-03 10:47:27.483502 7fcdffcdb700 10 log_queue is 6 last_log 210 sent 208 num 6 unsent 2 sending 2 -5266> 2013-04-03 10:47:27.483503 7fcdffcdb700 10 will send 2013-04-03 10:47:27.483302 mon.0 10.252.24.25:6789/0 208 : [INF] mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -5265> 2013-04-03 10:47:27.483507 7fcdffcdb700 10 will send 2013-04-03 10:47:27.483377 mon.0 10.252.24.25:6789/0 209 : [INF] osdmap e605: 5 osds: 5 up, 5 in -5264> 2013-04-03 10:47:27.483511 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x21fec00 -5263> 2013-04-03 10:47:27.483522 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5262> 2013-04-03 10:47:27.483548 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 create_pending v 2062 -5261> 2013-04-03 10:47:27.483569 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 AuthMonitor::on_active() -5260> 2013-04-03 10:47:27.483594 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos -5259> 2013-04-03 10:47:27.483608 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 -5258> 2013-04-03 10:47:27.483617 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -5257> 2013-04-03 10:47:27.483628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_log log(1 entries) v1 from mon.0 -5256> 2013-04-03 10:47:27.483646 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -5255> 2013-04-03 10:47:27.483657 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_log log(1 entries) v1 from mon.0 -5254> 2013-04-03 10:47:27.483667 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:22.267737 mon.0 10.252.24.25:6789/0 205 : [INF] mon.hotcpc9039 calling new monitor election -5253> 2013-04-03 10:47:27.483713 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140494 v187) v2 from mds.? 10.252.58.31:6800/10908 -5252> 2013-04-03 10:47:27.483723 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140494 v187) v2 -- ?+0 0x2145000 con 0x2118420 -5251> 2013-04-03 10:47:27.483754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 preprocess_query pg_stats(1 pgs tid 30464 v 605) v1 from osd.3 10.252.24.88:6801/17538 -5250> 2013-04-03 10:47:27.483761 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 PGMonitor::preprocess_pg_stats: no monitor session! -5249> 2013-04-03 10:47:27.483783 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 preprocess_query pg_stats(1 pgs tid 30465 v 605) v1 from osd.3 10.252.24.88:6801/17538 -5248> 2013-04-03 10:47:27.483789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 PGMonitor::preprocess_pg_stats: no monitor session! -5247> 2013-04-03 10:47:27.483806 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5246> 2013-04-03 10:47:27.483821 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 0 26 bytes epoch 5) v1 from osd.2 10.252.24.30:6801/5656 -5245> 2013-04-03 10:47:27.483834 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=26 -5244> 2013-04-03 10:47:27.483868 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6801/5656 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x1c16a00 con 0x3311c60 -5243> 2013-04-03 10:47:27.483878 7fcdffcdb700 0 -- 10.252.24.25:6789/0 submit_message auth_reply(proto 2 0 Success) v1 remote, 10.252.24.30:6801/5656, failed lossy con, dropping message 0x1c16a00 -5242> 2013-04-03 10:47:27.483903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 update_from_paxos -5241> 2013-04-03 10:47:27.483919 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 preprocess_query auth(proto 0 36 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/18040 -5240> 2013-04-03 10:47:27.483930 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=36 -5239> 2013-04-03 10:47:27.483942 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 AuthMonitor::assign_global_id m=auth(proto 0 36 bytes epoch 0) v1 mon=0/3 last_allocated=11896 max_global_id=11896 -5238> 2013-04-03 10:47:27.483953 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 next_global_id should be 11898 -5237> 2013-04-03 10:47:27.483962 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prepare_update auth(proto 0 36 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/18040 -5236> 2013-04-03 10:47:27.483974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 prep_auth() blob_size=36 -5235> 2013-04-03 10:47:27.483984 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 AuthMonitor::assign_global_id m=auth(proto 0 36 bytes epoch 0) v1 mon=0/3 last_allocated=11896 max_global_id=11896 -5234> 2013-04-03 10:47:27.483993 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 next_global_id should be 11898 -5233> 2013-04-03 10:47:27.484001 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 increasing max_global_id to 11996 -5232> 2013-04-03 10:47:27.484009 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 increasing global id, waitlisting message -5231> 2013-04-03 10:47:27.484056 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2061 encode_pending v 2062 -5230> 2013-04-03 10:47:27.484096 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427555) queue_proposal bl 174 bytes; ctx = 0x2c3ddd0 -5229> 2013-04-03 10:47:27.484104 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427555) propose_queued 427556 174 bytes -5228> 2013-04-03 10:47:27.485613 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427555 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3003280 -5227> 2013-04-03 10:47:27.485704 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos -5226> 2013-04-03 10:47:27.485720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 -5225> 2013-04-03 10:47:27.485732 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -5224> 2013-04-03 10:47:27.485743 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_log log(1 entries) v1 from mon.0 -5223> 2013-04-03 10:47:27.485764 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 nothing new -5222> 2013-04-03 10:47:27.485778 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b75600 con 0x1bac160 -5221> 2013-04-03 10:47:27.485797 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos -5220> 2013-04-03 10:47:27.485812 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 -5219> 2013-04-03 10:47:27.485821 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -5218> 2013-04-03 10:47:27.485831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_log log(2 entries) v1 from mon.0 -5217> 2013-04-03 10:47:27.485858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -5216> 2013-04-03 10:47:27.485870 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_log log(2 entries) v1 from mon.0 -5215> 2013-04-03 10:47:27.485880 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:22.267737 mon.0 10.252.24.25:6789/0 205 : [INF] mon.hotcpc9039 calling new monitor election -5214> 2013-04-03 10:47:27.485900 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:27.270635 mon.0 10.252.24.25:6789/0 206 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 -5213> 2013-04-03 10:47:27.485930 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2201000 con 0x1bac160 -5212> 2013-04-03 10:47:27.485950 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos -5211> 2013-04-03 10:47:27.485964 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 -5210> 2013-04-03 10:47:27.485973 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -5209> 2013-04-03 10:47:27.485983 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_log log(2 entries) v1 from mon.0 -5208> 2013-04-03 10:47:27.486006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -5207> 2013-04-03 10:47:27.486017 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_log log(2 entries) v1 from mon.0 -5206> 2013-04-03 10:47:27.486033 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:27.270635 mon.0 10.252.24.25:6789/0 206 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 -5205> 2013-04-03 10:47:27.486053 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:27.483218 mon.0 10.252.24.25:6789/0 207 : [INF] pgmap v29139: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 686B/s wr, 0op/s -5204> 2013-04-03 10:47:27.486081 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x209c600 con 0x1bac160 -5203> 2013-04-03 10:47:27.486101 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos -5202> 2013-04-03 10:47:27.486116 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 -5201> 2013-04-03 10:47:27.486125 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -5200> 2013-04-03 10:47:27.486135 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_log log(2 entries) v1 from mon.0 -5199> 2013-04-03 10:47:27.486158 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -5198> 2013-04-03 10:47:27.486168 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_log log(2 entries) v1 from mon.0 -5197> 2013-04-03 10:47:27.486178 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:27.483218 mon.0 10.252.24.25:6789/0 207 : [INF] pgmap v29139: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 686B/s wr, 0op/s -5196> 2013-04-03 10:47:27.486197 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:27.483302 mon.0 10.252.24.25:6789/0 208 : [INF] mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -5195> 2013-04-03 10:47:27.486226 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x21fec00 con 0x1bac160 -5194> 2013-04-03 10:47:27.486246 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos -5193> 2013-04-03 10:47:27.486260 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 -5192> 2013-04-03 10:47:27.486269 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -5191> 2013-04-03 10:47:27.486279 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_log log(2 entries) v1 from mon.0 -5190> 2013-04-03 10:47:27.486302 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -5189> 2013-04-03 10:47:27.486314 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_log log(2 entries) v1 from mon.0 -5188> 2013-04-03 10:47:27.486323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:27.483302 mon.0 10.252.24.25:6789/0 208 : [INF] mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -5187> 2013-04-03 10:47:27.486348 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:27.483377 mon.0 10.252.24.25:6789/0 209 : [INF] osdmap e605: 5 osds: 5 up, 5 in -5186> 2013-04-03 10:47:27.526187 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 7 ==== paxos(lease_ack lc 427555 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (4081426077 0 0) 0x3303780 con 0x208a840 -5185> 2013-04-03 10:47:27.526720 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 8 ==== forward(log(1 entries) v1) to leader v1 ==== 525+0+0 (440165160 0 0) 0x2cc58c0 con 0x208a840 -5184> 2013-04-03 10:47:27.526736 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mon.2 10.252.24.88:6789/0 via mon.2 10.252.24.88:6789/0 -5183> 2013-04-03 10:47:27.526744 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24ebe00 from 10.252.24.88:6789/0 -5182> 2013-04-03 10:47:27.526775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos -5181> 2013-04-03 10:47:27.526791 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 -5180> 2013-04-03 10:47:27.526801 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_query log(1 entries) v1 from mon.2 10.252.24.88:6789/0 -5179> 2013-04-03 10:47:27.526811 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 preprocess_log log(1 entries) v1 from mon.2 -5178> 2013-04-03 10:47:27.526824 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_update log(1 entries) v1 from mon.2 10.252.24.88:6789/0 -5177> 2013-04-03 10:47:27.526836 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 prepare_log log(1 entries) v1 from mon.2 -5176> 2013-04-03 10:47:27.526852 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 logging 2013-04-03 10:47:22.226658 mon.2 10.252.24.88:6789/0 1 : [INF] mon.hotcpc12533 calling new monitor election -5175> 2013-04-03 10:47:27.526882 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 9 ==== global_id (11896) v1 ==== 26+0+0 (2141558390 0 0) 0x2e8b700 con 0x208a840 -5174> 2013-04-03 10:47:27.538099 7fcdfddca700 2 -- 10.252.24.25:6789/0 >> 10.252.24.25:0/2614196857 pipe(0x3303000 sd=30 :6789 s=2 pgs=19586 cs=1 l=1).reader couldn't read tag, Success -5173> 2013-04-03 10:47:27.538120 7fcdfddca700 2 -- 10.252.24.25:6789/0 >> 10.252.24.25:0/2614196857 pipe(0x3303000 sd=30 :6789 s=2 pgs=19586 cs=1 l=1).fault 0: Success -5172> 2013-04-03 10:47:27.538165 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_handle_reset 0x3311840 10.252.24.25:0/2614196857 -5171> 2013-04-03 10:47:27.538173 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 reset/close on session client.11042 10.252.24.25:0/2614196857 -5170> 2013-04-03 10:47:27.538177 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 remove_session MonSession: client.11042 10.252.24.25:0/2614196857 is open client.11042 10.252.24.25:0/2614196857 -5169> 2013-04-03 10:47:27.538175 7fcdfdcc9700 2 -- 10.252.24.25:6789/0 >> 10.252.24.25:0/2151250553 pipe(0x2116c80 sd=31 :6789 s=2 pgs=22707 cs=1 l=1).reader couldn't read tag, Success -5168> 2013-04-03 10:47:27.538188 7fcdfdcc9700 2 -- 10.252.24.25:6789/0 >> 10.252.24.25:0/2151250553 pipe(0x2116c80 sd=31 :6789 s=2 pgs=22707 cs=1 l=1).fault 0: Success -5167> 2013-04-03 10:47:27.538224 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_handle_reset 0x33116e0 10.252.24.25:0/2151250553 -5166> 2013-04-03 10:47:27.538230 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 reset/close on session client.11045 10.252.24.25:0/2151250553 -5165> 2013-04-03 10:47:27.538233 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 remove_session MonSession: client.11045 10.252.24.25:0/2151250553 is open client.11045 10.252.24.25:0/2151250553 -5164> 2013-04-03 10:47:27.539433 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 10 ==== paxos(accept lc 427555 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (1720427458 0 0) 0x3303500 con 0x208a840 -5163> 2013-04-03 10:47:27.539745 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427556 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3303780 -5162> 2013-04-03 10:47:27.539777 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427556 fc 427522 pn 0 opn 0) v3 -- ?+0 0x32df780 -5161> 2013-04-03 10:47:27.539817 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos -5160> 2013-04-03 10:47:27.539853 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos version 2062 keys ver 2061 latest 2059 -5159> 2013-04-03 10:47:27.539863 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos key server version 2061 -5158> 2013-04-03 10:47:27.539880 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos() last_allocated_id=11896 max_global_id=11996 -5157> 2013-04-03 10:47:27.539896 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 create_pending v 2063 -5156> 2013-04-03 10:47:27.539928 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos -5155> 2013-04-03 10:47:27.539943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11802 10.252.24.30:0/517 -5154> 2013-04-03 10:47:27.539956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 prep_auth() blob_size=30 -5153> 2013-04-03 10:47:27.539965 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 no session, dropping -5152> 2013-04-03 10:47:27.539986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos -5151> 2013-04-03 10:47:27.540001 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11042 10.252.24.25:0/2614196857 -5150> 2013-04-03 10:47:27.540012 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 prep_auth() blob_size=30 -5149> 2013-04-03 10:47:27.540021 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 no session, dropping -5148> 2013-04-03 10:47:27.540041 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos -5147> 2013-04-03 10:47:27.540058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11045 10.252.24.25:0/2151250553 -5146> 2013-04-03 10:47:27.540070 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 prep_auth() blob_size=30 -5145> 2013-04-03 10:47:27.540078 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 no session, dropping -5144> 2013-04-03 10:47:27.540098 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos -5143> 2013-04-03 10:47:27.540113 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 preprocess_query auth(proto 0 26 bytes epoch 5) v1 from osd.0 10.252.58.31:6802/2682 -5142> 2013-04-03 10:47:27.540124 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 prep_auth() blob_size=26 -5141> 2013-04-03 10:47:27.540149 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6802/2682 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x2b78600 con 0x3311580 -5140> 2013-04-03 10:47:27.540178 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos -5139> 2013-04-03 10:47:27.540194 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11071 10.252.24.32:0/1396772501 -5138> 2013-04-03 10:47:27.540206 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 prep_auth() blob_size=30 -5137> 2013-04-03 10:47:27.540225 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.32:0/1396772501 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x2b78000 con 0x3311420 -5136> 2013-04-03 10:47:27.540257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 update_from_paxos -5135> 2013-04-03 10:47:27.540273 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 preprocess_query global_id (11896) v1 from mon.2 10.252.24.88:6789/0 -5134> 2013-04-03 10:47:27.540285 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 prepare_update global_id (11896) v1 from mon.2 10.252.24.88:6789/0 -5133> 2013-04-03 10:47:27.540297 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 AuthMonitor::prepare_global_id -5132> 2013-04-03 10:47:27.540305 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 increasing max_global_id to 12096 -5131> 2013-04-03 10:47:27.540706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2062 encode_pending v 2063 -5130> 2013-04-03 10:47:27.540746 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427556) queue_proposal bl 174 bytes; ctx = 0x330b000 -5129> 2013-04-03 10:47:27.540755 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427556) propose_queued 427557 174 bytes -5128> 2013-04-03 10:47:27.549368 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427556 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3003000 -5127> 2013-04-03 10:47:27.549427 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11071 10.252.24.32:0/1396772501 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (538169822 0 0) 0x287ea00 con 0x3311420 -5126> 2013-04-03 10:47:27.549446 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.0 10.252.58.31:6802/2682 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (1547225235 0 0) 0x2117c00 con 0x3311580 -5125> 2013-04-03 10:47:27.551471 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 11 ==== paxos(lease_ack lc 427556 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (908041699 0 0) 0x3003000 con 0x208a840 -5124> 2013-04-03 10:47:27.589771 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 12 ==== paxos(accept lc 427556 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (2519637452 0 0) 0x32df780 con 0x208a840 -5123> 2013-04-03 10:47:27.590073 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427557 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3003000 -5122> 2013-04-03 10:47:27.590105 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427557 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3303500 -5121> 2013-04-03 10:47:27.590148 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -5120> 2013-04-03 10:47:27.590178 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos version 2063 keys ver 2062 latest 2059 -5119> 2013-04-03 10:47:27.590187 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos key server version 2062 -5118> 2013-04-03 10:47:27.590202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos() last_allocated_id=11896 max_global_id=12096 -5117> 2013-04-03 10:47:27.590218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 create_pending v 2064 -5116> 2013-04-03 10:47:27.590248 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -5115> 2013-04-03 10:47:27.590261 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 36 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/18040 -5114> 2013-04-03 10:47:27.590272 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=36 -5113> 2013-04-03 10:47:27.590280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 AuthMonitor::assign_global_id m=auth(proto 0 36 bytes epoch 0) v1 mon=0/3 last_allocated=11896 max_global_id=12096 -5112> 2013-04-03 10:47:27.590289 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 next_global_id should be 11898 -5111> 2013-04-03 10:47:27.590298 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- auth_reply(proto 0 -22 Invalid argument) v1 -- ?+0 0x2b78400 con 0x3311b00 -5110> 2013-04-03 10:47:27.590304 7fcdffcdb700 0 -- 10.252.24.25:6789/0 submit_message auth_reply(proto 0 -22 Invalid argument) v1 remote, 10.252.24.88:6800/18040, failed lossy con, dropping message 0x2b78400 -5109> 2013-04-03 10:47:27.590325 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -5108> 2013-04-03 10:47:27.590339 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.11071 10.252.24.32:0/1396772501 -5107> 2013-04-03 10:47:27.590348 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 -5106> 2013-04-03 10:47:27.590476 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.32:0/1396772501 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x2201200 con 0x3311420 -5105> 2013-04-03 10:47:27.590508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -5104> 2013-04-03 10:47:27.590522 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from osd.0 10.252.58.31:6802/2682 -5103> 2013-04-03 10:47:27.590533 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 -5102> 2013-04-03 10:47:27.590645 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6802/2682 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x287ea00 con 0x3311580 -5101> 2013-04-03 10:47:27.590658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 AuthMonitor::on_active() -5100> 2013-04-03 10:47:27.590715 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos -5099> 2013-04-03 10:47:27.590730 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 -5098> 2013-04-03 10:47:27.590747 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -5097> 2013-04-03 10:47:27.591169 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11071 10.252.24.32:0/1396772501 3 ==== mon_subscribe({mdsmap=188+,monmap=6+}) v2 ==== 42+0+0 (435898880 0 0) 0x2e8b540 con 0x3311420 -5096> 2013-04-03 10:47:27.591193 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=188+,monmap=6+}) v2 -5095> 2013-04-03 10:47:27.591203 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -5094> 2013-04-03 10:47:27.591205 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11071 10.252.24.32:0/1396772501 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8b700 -5093> 2013-04-03 10:47:27.591215 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.0 10.252.58.31:6802/2682 3 ==== mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=604}) v2 ==== 69+0+0 (4250075019 0 0) 0x2e8b8c0 con 0x3311580 -5092> 2013-04-03 10:47:27.591222 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=604}) v2 -5091> 2013-04-03 10:47:27.591225 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -5090> 2013-04-03 10:47:27.591231 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e605 send_incremental [604..605] to osd.0 10.252.58.31:6802/2682 -5089> 2013-04-03 10:47:27.591250 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 build_incremental [604..605] -5088> 2013-04-03 10:47:27.591277 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.0 10.252.58.31:6802/2682 -- osd_map(604..605 src has 104..605) v3 -- ?+0 0x2117c00 -5087> 2013-04-03 10:47:27.591288 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.0 10.252.58.31:6802/2682 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8b540 -5086> 2013-04-03 10:47:27.591302 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.0 10.252.58.31:6802/2682 4 ==== mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=604}) v2 ==== 69+0+0 (4250075019 0 0) 0x2e8bc40 con 0x3311580 -5085> 2013-04-03 10:47:27.591312 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=604}) v2 -5084> 2013-04-03 10:47:27.591317 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -5083> 2013-04-03 10:47:27.591320 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e605 send_incremental [604..605] to osd.0 10.252.58.31:6802/2682 -5082> 2013-04-03 10:47:27.591335 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 build_incremental [604..605] -5081> 2013-04-03 10:47:27.591359 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.0 10.252.58.31:6802/2682 -- osd_map(604..605 src has 104..605) v3 -- ?+0 0x2b78400 -5080> 2013-04-03 10:47:27.591368 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.0 10.252.58.31:6802/2682 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8b8c0 -5079> 2013-04-03 10:47:27.600598 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 212 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140495 v187) v2 ==== 289+0+0 (3132436153 0 0) 0x20e4080 con 0x2118420 -5078> 2013-04-03 10:47:27.600632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140495 v187) v2 from mds.? 10.252.58.31:6800/10908 -5077> 2013-04-03 10:47:27.600642 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140495 v187) v2 -- ?+0 0x2a832c0 con 0x2118420 -5076> 2013-04-03 10:47:27.601794 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 13 ==== paxos(lease_ack lc 427557 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3868984203 0 0) 0x3303500 con 0x208a840 -5075> 2013-04-03 10:47:27.601841 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos -5074> 2013-04-03 10:47:27.601859 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 -5073> 2013-04-03 10:47:27.601874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -5072> 2013-04-03 10:47:27.845894 7fcdfdcc9700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3062500 sd=21 :6789 s=0 pgs=0 cs=0 l=0).accept sd=21 10.252.24.88:33190/0 -5071> 2013-04-03 10:47:27.846229 7fcdfdcc9700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.88:6804/18376 osd protocol 0 -5070> 2013-04-03 10:47:27.846558 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 1 ==== auth(proto 0 26 bytes epoch 0) v1 ==== 56+0+0 (3146543200 0 0) 0x287ea00 con 0x33112c0 -5069> 2013-04-03 10:47:27.846590 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -5068> 2013-04-03 10:47:27.846596 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: osd.3 10.252.24.88:6804/18376 is open for osd.3 10.252.24.88:6804/18376 -5067> 2013-04-03 10:47:27.846601 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -5066> 2013-04-03 10:47:27.846644 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -5065> 2013-04-03 10:47:27.846661 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 26 bytes epoch 0) v1 from osd.3 10.252.24.88:6804/18376 -5064> 2013-04-03 10:47:27.846673 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=26 -5063> 2013-04-03 10:47:27.846687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 AuthMonitor::assign_global_id m=auth(proto 0 26 bytes epoch 0) v1 mon=0/3 last_allocated=11898 max_global_id=12096 -5062> 2013-04-03 10:47:27.846699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 next_global_id should be 11901 -5061> 2013-04-03 10:47:27.846711 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6804/18376 -- mon_map v1 -- ?+0 0x2b74e00 con 0x33112c0 -5060> 2013-04-03 10:47:27.846741 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6804/18376 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x209c200 con 0x33112c0 -5059> 2013-04-03 10:47:27.847467 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (970034522 0 0) 0x209c200 con 0x33112c0 -5058> 2013-04-03 10:47:27.847515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -5057> 2013-04-03 10:47:27.847532 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 32 bytes epoch 0) v1 from osd.3 10.252.24.88:6804/18376 -5056> 2013-04-03 10:47:27.847544 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=32 -5055> 2013-04-03 10:47:27.847653 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6804/18376 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x287ea00 con 0x33112c0 -5054> 2013-04-03 10:47:27.848252 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (2183087255 0 0) 0x2b74e00 con 0x33112c0 -5053> 2013-04-03 10:47:27.848301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -5052> 2013-04-03 10:47:27.848317 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 165 bytes epoch 0) v1 from osd.3 10.252.24.88:6804/18376 -5051> 2013-04-03 10:47:27.848328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=165 -5050> 2013-04-03 10:47:27.848476 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6804/18376 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x209c200 con 0x33112c0 -5049> 2013-04-03 10:47:27.849083 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 4 ==== mon_subscribe({monmap=0+}) v2 ==== 23+0+0 (1620593354 0 0) 0x2e8b540 con 0x33112c0 -5048> 2013-04-03 10:47:27.849105 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=0+}) v2 -5047> 2013-04-03 10:47:27.849113 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 0 have 5 -5046> 2013-04-03 10:47:27.849117 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6804/18376 -- mon_map v1 -- ?+0 0x2b74e00 con 0x33112c0 -5045> 2013-04-03 10:47:27.849129 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.3 10.252.24.88:6804/18376 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8bc40 -5044> 2013-04-03 10:47:27.849138 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 5 ==== mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 ==== 50+0+0 (2151744701 0 0) 0x2e8be00 con 0x33112c0 -5043> 2013-04-03 10:47:27.849146 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -5042> 2013-04-03 10:47:27.849149 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 0 have 5 -5041> 2013-04-03 10:47:27.849152 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6804/18376 -- mon_map v1 -- ?+0 0x209ce00 con 0x33112c0 -5040> 2013-04-03 10:47:27.849160 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.3 10.252.24.88:6804/18376 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8b540 -5039> 2013-04-03 10:47:27.849167 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 6 ==== auth(proto 2 2 bytes epoch 0) v1 ==== 32+0+0 (2260890001 0 0) 0x2201200 con 0x33112c0 -5038> 2013-04-03 10:47:27.849193 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -5037> 2013-04-03 10:47:27.849209 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 2 bytes epoch 0) v1 from osd.3 10.252.24.88:6804/18376 -5036> 2013-04-03 10:47:27.849220 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=2 -5035> 2013-04-03 10:47:27.849256 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6804/18376 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x25f0200 con 0x33112c0 -5034> 2013-04-03 10:47:27.850350 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 7 ==== mon_get_version(what=osdmap handle=1) v1 ==== 18+0+0 (4194021778 0 0) 0x2e8b8c0 con 0x33112c0 -5033> 2013-04-03 10:47:27.850372 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_get_version mon_get_version(what=osdmap handle=1) v1 -5032> 2013-04-03 10:47:27.850395 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.3 10.252.24.88:6804/18376 -- mon_check_map_ack(handle=1 version=605) v2 -- ?+0 0x2e8be00 -5031> 2013-04-03 10:47:27.850967 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 8 ==== osd_boot(osd.3 booted 0 v605) v3 ==== 626+0+0 (746860765 0 0) 0x2071c00 con 0x33112c0 -5030> 2013-04-03 10:47:27.850999 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 preprocess_query osd_boot(osd.3 booted 0 v605) v3 from osd.3 10.252.24.88:6804/18376 -5029> 2013-04-03 10:47:27.851007 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 preprocess_boot from osd.3 10.252.24.88:6804/18376 -5028> 2013-04-03 10:47:27.851011 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e605 prepare_update osd_boot(osd.3 booted 0 v605) v3 from osd.3 10.252.24.88:6804/18376 -5027> 2013-04-03 10:47:27.851014 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e605 prepare_boot from osd.3 10.252.24.88:6804/18376 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.3 0b7e2707-d3a7-4f74-93fa-0686f3188d92 e605 [104,605] lci=[576,605]) cluster_addr 10.252.24.88:6805/18376 hb_addr 10.252.24.88:6806/18376 -5026> 2013-04-03 10:47:27.851023 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e605 prepare_boot was up, first marking down osd.3 10.252.24.88:6801/17538 -5025> 2013-04-03 10:47:27.851027 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 should_propose -5024> 2013-04-03 10:47:28.384969 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29139 check_down_pgs -5023> 2013-04-03 10:47:28.384993 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29139 v29139: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 686B/s wr, 0op/s -5022> 2013-04-03 10:47:28.385022 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e187 e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -5021> 2013-04-03 10:47:28.385047 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 e605: 5 osds: 5 up, 5 in -5020> 2013-04-03 10:47:28.385083 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 min_last_epoch_clean 593 -5019> 2013-04-03 10:47:28.385113 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos -5018> 2013-04-03 10:47:28.385132 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394127 update_from_paxos version 394127 summary v 394127 -5017> 2013-04-03 10:47:28.385141 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394127 log -5016> 2013-04-03 10:47:28.385158 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -5015> 2013-04-03 10:47:28.385180 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -5014> 2013-04-03 10:47:28.477267 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394127 encode_pending v394128 -5013> 2013-04-03 10:47:28.477303 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427557) queue_proposal bl 1537 bytes; ctx = 0x2c3de30 -5012> 2013-04-03 10:47:28.477311 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427557) propose_queued 427558 1537 bytes -5011> 2013-04-03 10:47:28.478941 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427557 fc 0 pn 17700 opn 0) v3 -- ?+0 0x300c280 -5010> 2013-04-03 10:47:28.559002 7fcdfddca700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x20f4a00 sd=23 :6789 s=0 pgs=0 cs=0 l=0).accept sd=23 10.252.24.25:41776/0 -5009> 2013-04-03 10:47:28.559030 7fcdfddca700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.25:6805/3572 osd protocol 0 -5008> 2013-04-03 10:47:28.559116 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 1 ==== auth(proto 0 26 bytes epoch 5) v1 ==== 56+0+0 (1874204925 0 0) 0x287ea00 con 0x3311160 -5007> 2013-04-03 10:47:28.559128 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -5006> 2013-04-03 10:47:28.559133 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: osd.1 10.252.24.25:6805/3572 is open for osd.1 10.252.24.25:6805/3572 -5005> 2013-04-03 10:47:28.559138 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -5004> 2013-04-03 10:47:28.559162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -5003> 2013-04-03 10:47:28.559178 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 26 bytes epoch 5) v1 from osd.1 10.252.24.25:6805/3572 -5002> 2013-04-03 10:47:28.559190 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=26 -5001> 2013-04-03 10:47:28.559221 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x1c16a00 con 0x3311160 -5000> 2013-04-03 10:47:28.559399 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (2074475155 0 0) 0x2b74e00 con 0x3311160 -4999> 2013-04-03 10:47:28.559424 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -4998> 2013-04-03 10:47:28.559441 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from osd.1 10.252.24.25:6805/3572 -4997> 2013-04-03 10:47:28.559453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 -4996> 2013-04-03 10:47:28.559580 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x287ea00 con 0x3311160 -4995> 2013-04-03 10:47:28.559748 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 3 ==== mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=603}) v2 ==== 69+0+0 (976368017 0 0) 0x2e8bc40 con 0x3311160 -4994> 2013-04-03 10:47:28.559759 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=603}) v2 -4993> 2013-04-03 10:47:28.559766 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -4992> 2013-04-03 10:47:28.559771 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e605 send_incremental [603..605] to osd.1 10.252.24.25:6805/3572 -4991> 2013-04-03 10:47:28.559790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 build_incremental [603..605] -4990> 2013-04-03 10:47:28.559825 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.1 10.252.24.25:6805/3572 -- osd_map(603..605 src has 104..605) v3 -- ?+0 0x2b74e00 -4989> 2013-04-03 10:47:28.559836 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.1 10.252.24.25:6805/3572 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8b8c0 -4988> 2013-04-03 10:47:28.559849 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 4 ==== pg_stats(1 pgs tid 69 v 605) v1 ==== 481+0+0 (3004635642 0 0) 0x24f8240 con 0x3311160 -4987> 2013-04-03 10:47:28.559880 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 preprocess_query pg_stats(1 pgs tid 69 v 605) v1 from osd.1 10.252.24.25:6805/3572 -4986> 2013-04-03 10:47:28.559901 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 prepare_update pg_stats(1 pgs tid 69 v 605) v1 from osd.1 10.252.24.25:6805/3572 -4985> 2013-04-03 10:47:28.559906 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 prepare_pg_stats pg_stats(1 pgs tid 69 v 605) v1 from osd.1 -4984> 2013-04-03 10:47:28.559912 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29139 got osd.1 osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -4983> 2013-04-03 10:47:28.559933 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 5 ==== mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=603}) v2 ==== 69+0+0 (976368017 0 0) 0x2224380 con 0x3311160 -4982> 2013-04-03 10:47:28.559942 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=603}) v2 -4981> 2013-04-03 10:47:28.559946 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -4980> 2013-04-03 10:47:28.559949 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e605 send_incremental [603..605] to osd.1 10.252.24.25:6805/3572 -4979> 2013-04-03 10:47:28.559963 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e605 build_incremental [603..605] -4978> 2013-04-03 10:47:28.559995 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.1 10.252.24.25:6805/3572 -- osd_map(603..605 src has 104..605) v3 -- ?+0 0x2b74400 -4977> 2013-04-03 10:47:28.560004 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.1 10.252.24.25:6805/3572 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8b000 -4976> 2013-04-03 10:47:28.589946 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29139 encode_pending v 29140 -4975> 2013-04-03 10:47:28.589980 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427522..427557) queue_proposal bl 650 bytes; ctx = 0x2c3df10 -4974> 2013-04-03 10:47:28.589987 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427522..427557) propose_new_value not active; proposal queued -4973> 2013-04-03 10:47:28.590039 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e605 encode_pending e 606 -4972> 2013-04-03 10:47:28.590043 7fce004dc700 2 mon.hotcpc9039@0(leader).osd e605 osd.3 DOWN -4971> 2013-04-03 10:47:28.590072 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427522..427557) queue_proposal bl 290 bytes; ctx = 0x2c3df30 -4970> 2013-04-03 10:47:28.590079 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427522..427557) propose_new_value not active; proposal queued -4969> 2013-04-03 10:47:28.601200 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 14 ==== paxos(accept lc 427557 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (3135003404 0 0) 0x300c280 con 0x208a840 -4968> 2013-04-03 10:47:28.601490 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427558 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3250280 -4967> 2013-04-03 10:47:28.601510 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427558 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250c80 -4966> 2013-04-03 10:47:28.601552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos -4965> 2013-04-03 10:47:28.601705 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos version 394128 summary v 394127 -4964> 2013-04-03 10:47:28.601720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos latest full 393798 -4963> 2013-04-03 10:47:28.601738 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 update_from_paxos applying incremental log 394128 2013-04-03 10:47:22.226658 mon.2 10.252.24.88:6789/0 1 : [INF] mon.hotcpc12533 calling new monitor election -4962> 2013-04-03 10:47:28.601765 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 update_from_paxos applying incremental log 394128 2013-04-03 10:47:22.267737 mon.0 10.252.24.25:6789/0 205 : [INF] mon.hotcpc9039 calling new monitor election -4961> 2013-04-03 10:47:28.601785 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 update_from_paxos applying incremental log 394128 2013-04-03 10:47:27.270635 mon.0 10.252.24.25:6789/0 206 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 -4960> 2013-04-03 10:47:28.602145 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 update_from_paxos applying incremental log 394128 2013-04-03 10:47:27.483218 mon.0 10.252.24.25:6789/0 207 : [INF] pgmap v29139: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 686B/s wr, 0op/s -4959> 2013-04-03 10:47:28.602167 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 update_from_paxos applying incremental log 394128 2013-04-03 10:47:27.483302 mon.0 10.252.24.25:6789/0 208 : [INF] mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -4958> 2013-04-03 10:47:28.602186 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 update_from_paxos applying incremental log 394128 2013-04-03 10:47:27.483377 mon.0 10.252.24.25:6789/0 209 : [INF] osdmap e605: 5 osds: 5 up, 5 in -4957> 2013-04-03 10:47:28.602224 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 check_subs -4956> 2013-04-03 10:47:28.602259 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 create_pending v 394129 -4955> 2013-04-03 10:47:28.602282 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 _updated_log for mon.0 10.252.24.25:6789/0 -4954> 2013-04-03 10:47:28.602294 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 205) v1 -- ?+0 0x2224380 con 0x1bac160 -4953> 2013-04-03 10:47:28.602301 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 _updated_log for mon.0 10.252.24.25:6789/0 -4952> 2013-04-03 10:47:28.602314 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 206) v1 -- ?+0 0x33068c0 con 0x1bac160 -4951> 2013-04-03 10:47:28.602319 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 _updated_log for mon.0 10.252.24.25:6789/0 -4950> 2013-04-03 10:47:28.602331 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 207) v1 -- ?+0 0x28a5700 con 0x1bac160 -4949> 2013-04-03 10:47:28.602336 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 _updated_log for mon.0 10.252.24.25:6789/0 -4948> 2013-04-03 10:47:28.602348 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 208) v1 -- ?+0 0x28a5c40 con 0x1bac160 -4947> 2013-04-03 10:47:28.602353 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 _updated_log for mon.0 10.252.24.25:6789/0 -4946> 2013-04-03 10:47:28.602367 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 209) v1 -- ?+0 0x3306e00 con 0x1bac160 -4945> 2013-04-03 10:47:28.602372 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394128 _updated_log for mon.2 10.252.24.88:6789/0 -4944> 2013-04-03 10:47:28.602384 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(log(last 1) v1 tid 1) v2 -- ?+0 0x20ad900 con 0x208a840 -4943> 2013-04-03 10:47:28.602410 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427558) propose_queued 427559 650 bytes -4942> 2013-04-03 10:47:28.603600 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427558 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2fe4c80 -4941> 2013-04-03 10:47:28.603622 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 205) v1 ==== 0+0+0 (0 0 0) 0x2224380 con 0x1bac160 -4940> 2013-04-03 10:47:28.603630 7fcdffcdb700 10 handle_log_ack log(last 205) v1 -4939> 2013-04-03 10:47:28.603632 7fcdffcdb700 10 logged 2013-04-03 10:47:22.267737 mon.0 10.252.24.25:6789/0 205 : [INF] mon.hotcpc9039 calling new monitor election -4938> 2013-04-03 10:47:28.603639 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 206) v1 ==== 0+0+0 (0 0 0) 0x33068c0 con 0x1bac160 -4937> 2013-04-03 10:47:28.603998 7fcdffcdb700 10 handle_log_ack log(last 206) v1 -4936> 2013-04-03 10:47:28.603999 7fcdffcdb700 10 logged 2013-04-03 10:47:27.270635 mon.0 10.252.24.25:6789/0 206 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,2 -4935> 2013-04-03 10:47:28.604005 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 207) v1 ==== 0+0+0 (0 0 0) 0x28a5700 con 0x1bac160 -4934> 2013-04-03 10:47:28.604011 7fcdffcdb700 10 handle_log_ack log(last 207) v1 -4933> 2013-04-03 10:47:28.604012 7fcdffcdb700 10 logged 2013-04-03 10:47:27.483218 mon.0 10.252.24.25:6789/0 207 : [INF] pgmap v29139: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 686B/s wr, 0op/s -4932> 2013-04-03 10:47:28.604017 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 208) v1 ==== 0+0+0 (0 0 0) 0x28a5c40 con 0x1bac160 -4931> 2013-04-03 10:47:28.604023 7fcdffcdb700 10 handle_log_ack log(last 208) v1 -4930> 2013-04-03 10:47:28.604025 7fcdffcdb700 10 logged 2013-04-03 10:47:27.483302 mon.0 10.252.24.25:6789/0 208 : [INF] mdsmap e187: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -4929> 2013-04-03 10:47:28.604030 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 209) v1 ==== 0+0+0 (0 0 0) 0x3306e00 con 0x1bac160 -4928> 2013-04-03 10:47:28.604036 7fcdffcdb700 10 handle_log_ack log(last 209) v1 -4927> 2013-04-03 10:47:28.604037 7fcdffcdb700 10 logged 2013-04-03 10:47:27.483377 mon.0 10.252.24.25:6789/0 209 : [INF] osdmap e605: 5 osds: 5 up, 5 in -4926> 2013-04-03 10:47:28.626616 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 15 ==== paxos(lease_ack lc 427558 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1191558493 0 0) 0x2fe4c80 con 0x208a840 -4925> 2013-04-03 10:47:28.638464 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 16 ==== paxos(accept lc 427558 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (1784706665 0 0) 0x3250280 con 0x208a840 -4924> 2013-04-03 10:47:28.638794 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427559 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2fe4c80 -4923> 2013-04-03 10:47:28.638820 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427559 fc 427522 pn 0 opn 0) v3 -- ?+0 0x300c280 -4922> 2013-04-03 10:47:28.638901 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29139 update_from_paxos applying incremental 29140 -4921> 2013-04-03 10:47:28.638922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 v29140: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 899B/s wr, 0op/s -4920> 2013-04-03 10:47:28.638940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 send_pg_creates to 0 pgs -4919> 2013-04-03 10:47:28.638943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 update_logger -4918> 2013-04-03 10:47:28.638955 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 create_pending v 29141 -4917> 2013-04-03 10:47:28.638964 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29140 _updated_stats for osd.1 10.252.24.25:6805/3572 -4916> 2013-04-03 10:47:28.638970 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(1 pgs tid 69) v1 -- ?+0 0x2e8bc40 con 0x3311160 -4915> 2013-04-03 10:47:28.638981 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 check_osd_map already seen 605 >= 605 -4914> 2013-04-03 10:47:28.638983 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 update_logger -4913> 2013-04-03 10:47:28.638993 7fcdffcdb700 0 log [INF] : pgmap v29140: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 899B/s wr, 0op/s -4912> 2013-04-03 10:47:28.638994 7fcdffcdb700 10 send_log to self -4911> 2013-04-03 10:47:28.638995 7fcdffcdb700 10 log_queue is 2 last_log 211 sent 209 num 2 unsent 2 sending 2 -4910> 2013-04-03 10:47:28.638997 7fcdffcdb700 10 will send 2013-04-03 10:47:27.483501 mon.0 10.252.24.25:6789/0 210 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -4909> 2013-04-03 10:47:28.639002 7fcdffcdb700 10 will send 2013-04-03 10:47:28.638994 mon.0 10.252.24.25:6789/0 211 : [INF] pgmap v29140: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 899B/s wr, 0op/s -4908> 2013-04-03 10:47:28.639006 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x209c200 -4907> 2013-04-03 10:47:28.639032 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427559) propose_queued 427560 290 bytes -4906> 2013-04-03 10:47:28.639362 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427559 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2fe4500 -4905> 2013-04-03 10:47:28.639395 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x209c200 con 0x1bac160 -4904> 2013-04-03 10:47:28.639438 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos -4903> 2013-04-03 10:47:28.639454 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos version 394128 summary v 394128 -4902> 2013-04-03 10:47:28.639462 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4901> 2013-04-03 10:47:28.639471 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 preprocess_log log(2 entries) v1 from mon.0 -4900> 2013-04-03 10:47:28.639501 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4899> 2013-04-03 10:47:28.639515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 prepare_log log(2 entries) v1 from mon.0 -4898> 2013-04-03 10:47:28.639525 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 logging 2013-04-03 10:47:27.483501 mon.0 10.252.24.25:6789/0 210 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -4897> 2013-04-03 10:47:28.639544 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 logging 2013-04-03 10:47:28.638994 mon.0 10.252.24.25:6789/0 211 : [INF] pgmap v29140: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 899B/s wr, 0op/s -4896> 2013-04-03 10:47:28.663917 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 17 ==== paxos(lease_ack lc 427559 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (4013390661 0 0) 0x2fe4500 con 0x208a840 -4895> 2013-04-03 10:47:28.677128 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 18 ==== paxos(accept lc 427559 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (3005094146 0 0) 0x2fe4c80 con 0x208a840 -4894> 2013-04-03 10:47:28.677394 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427560 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2fe4500 -4893> 2013-04-03 10:47:28.677411 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427560 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250280 -4892> 2013-04-03 10:47:28.677758 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e605 update_from_paxos applying incremental 606 -4891> 2013-04-03 10:47:28.677803 7fcdffcdb700 1 mon.hotcpc9039@0(leader).osd e606 e606: 5 osds: 4 up, 5 in -4890> 2013-04-03 10:47:28.678888 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 adding osd.3 to down_pending_out map -4889> 2013-04-03 10:47:28.678942 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 check_osd_map applying osdmap e606 to pg_map -4888> 2013-04-03 10:47:28.678961 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 register_new_pgs checking pg pools for osdmap epoch 606, last_pg_scan 341 -4887> 2013-04-03 10:47:28.678966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 -4886> 2013-04-03 10:47:28.678970 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -4885> 2013-04-03 10:47:28.678972 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -4884> 2013-04-03 10:47:28.678974 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 -4883> 2013-04-03 10:47:28.678977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 -4882> 2013-04-03 10:47:28.678986 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 -4881> 2013-04-03 10:47:28.678989 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 -4880> 2013-04-03 10:47:28.678994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 -4879> 2013-04-03 10:47:28.678996 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 -4878> 2013-04-03 10:47:28.679012 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs -4877> 2013-04-03 10:47:28.679014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 check_down_pgs -4876> 2013-04-03 10:47:28.679015 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 8.4 stale with acting [3,2] -4875> 2013-04-03 10:47:28.679018 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.e stale with acting [3,2] -4874> 2013-04-03 10:47:28.679021 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.c stale with acting [3,2] -4873> 2013-04-03 10:47:28.679023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 8.2 stale with acting [3,2] -4872> 2013-04-03 10:47:28.679026 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.8 stale with acting [3,2] -4871> 2013-04-03 10:47:28.679028 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.b stale with acting [3,2] -4870> 2013-04-03 10:47:28.679030 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.a stale with acting [3,2] -4869> 2013-04-03 10:47:28.679033 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.a stale with acting [3,2] -4868> 2013-04-03 10:47:28.679035 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.9 stale with acting [3,2] -4867> 2013-04-03 10:47:28.679037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 6.0 stale with acting [3,0] -4866> 2013-04-03 10:47:28.679040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 4.2 stale with acting [3,0] -4865> 2013-04-03 10:47:28.679042 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.4 stale with acting [3,0] -4864> 2013-04-03 10:47:28.679044 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.6 stale with acting [3,0] -4863> 2013-04-03 10:47:28.679047 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 7.3 stale with acting [3,2] -4862> 2013-04-03 10:47:28.679049 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 5.1 stale with acting [3,0] -4861> 2013-04-03 10:47:28.679055 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 4.0 stale with acting [3,2] -4860> 2013-04-03 10:47:28.679057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 3.7 stale with acting [3,2] -4859> 2013-04-03 10:47:28.679060 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.5 stale with acting [3,0] -4858> 2013-04-03 10:47:28.679062 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.4 stale with acting [3,2] -4857> 2013-04-03 10:47:28.679064 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 7.5 stale with acting [3,2] -4856> 2013-04-03 10:47:28.679066 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 6.4 stale with acting [3,2] -4855> 2013-04-03 10:47:28.679068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 5.7 stale with acting [3,2] -4854> 2013-04-03 10:47:28.679071 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 4.6 stale with acting [3,2] -4853> 2013-04-03 10:47:28.679073 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 3.1 stale with acting [3,2] -4852> 2013-04-03 10:47:28.679075 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.0 stale with acting [3,0] -4851> 2013-04-03 10:47:28.679077 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.3 stale with acting [3,2] -4850> 2013-04-03 10:47:28.679080 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.2 stale with acting [3,0] -4849> 2013-04-03 10:47:28.679082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 6.6 stale with acting [3,2] -4848> 2013-04-03 10:47:28.679084 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 5.5 stale with acting [3,2] -4847> 2013-04-03 10:47:28.679086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 3.3 stale with acting [3,0] -4846> 2013-04-03 10:47:28.679089 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.2 stale with acting [3,2] -4845> 2013-04-03 10:47:28.679091 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.1 stale with acting [3,0] -4844> 2013-04-03 10:47:28.679094 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.3c stale with acting [3,2] -4843> 2013-04-03 10:47:28.679097 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.3b stale with acting [3,2] -4842> 2013-04-03 10:47:28.679102 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.3a stale with acting [3,2] -4841> 2013-04-03 10:47:28.679105 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.31 stale with acting [3,0] -4840> 2013-04-03 10:47:28.679107 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.32 stale with acting [3,0] -4839> 2013-04-03 10:47:28.679109 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.33 stale with acting [3,0] -4838> 2013-04-03 10:47:28.679111 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.30 stale with acting [3,2] -4837> 2013-04-03 10:47:28.679114 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.32 stale with acting [3,2] -4836> 2013-04-03 10:47:28.679116 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.31 stale with acting [3,2] -4835> 2013-04-03 10:47:28.679119 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.24 stale with acting [3,0] -4834> 2013-04-03 10:47:28.679122 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.23 stale with acting [3,0] -4833> 2013-04-03 10:47:28.679124 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.22 stale with acting [3,0] -4832> 2013-04-03 10:47:28.679127 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.1c stale with acting [3,1] -4831> 2013-04-03 10:47:28.679129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.1e stale with acting [3,1] -4830> 2013-04-03 10:47:28.679131 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.1c stale with acting [3,2] -4829> 2013-04-03 10:47:28.679133 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.1d stale with acting [3,2] -4828> 2013-04-03 10:47:28.679135 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.1d stale with acting [3,1] -4827> 2013-04-03 10:47:28.679138 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.1b stale with acting [3,4] -4826> 2013-04-03 10:47:28.679140 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.1a stale with acting [3,4] -4825> 2013-04-03 10:47:28.679142 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.19 stale with acting [3,4] -4824> 2013-04-03 10:47:28.679144 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.1a stale with acting [3,0] -4823> 2013-04-03 10:47:28.679150 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.18 stale with acting [3,0] -4822> 2013-04-03 10:47:28.679152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.1b stale with acting [3,2] -4821> 2013-04-03 10:47:28.679154 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.19 stale with acting [3,0] -4820> 2013-04-03 10:47:28.679157 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.16 stale with acting [3,1] -4819> 2013-04-03 10:47:28.679159 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.14 stale with acting [3,1] -4818> 2013-04-03 10:47:28.679161 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.15 stale with acting [3,4] -4817> 2013-04-03 10:47:28.679164 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.14 stale with acting [3,4] -4816> 2013-04-03 10:47:28.679166 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.14 stale with acting [3,1] -4815> 2013-04-03 10:47:28.679168 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.15 stale with acting [3,1] -4814> 2013-04-03 10:47:28.679170 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.13 stale with acting [3,4] -4813> 2013-04-03 10:47:28.679173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.12 stale with acting [3,4] -4812> 2013-04-03 10:47:28.679175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.11 stale with acting [3,4] -4811> 2013-04-03 10:47:28.679177 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.12 stale with acting [3,0] -4810> 2013-04-03 10:47:28.679179 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.13 stale with acting [3,1] -4809> 2013-04-03 10:47:28.679181 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.10 stale with acting [3,0] -4808> 2013-04-03 10:47:28.679184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.13 stale with acting [3,4] -4807> 2013-04-03 10:47:28.679186 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 0.10 stale with acting [3,2] -4806> 2013-04-03 10:47:28.679188 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.11 stale with acting [3,0] -4805> 2013-04-03 10:47:28.679190 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 2.12 stale with acting [3,1] -4804> 2013-04-03 10:47:28.679196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 marking pg 1.f stale with acting [3,2] -4803> 2013-04-03 10:47:28.679251 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 encode_pending v 29141 -4802> 2013-04-03 10:47:28.679454 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427560) queue_proposal bl 27444 bytes; ctx = 0x2c3df20 -4801> 2013-04-03 10:47:28.679461 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427560) propose_queued 427561 27444 bytes -4800> 2013-04-03 10:47:28.683709 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427560 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2fe4280 -4799> 2013-04-03 10:47:28.683727 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29140 send_pg_creates to 0 pgs -4798> 2013-04-03 10:47:28.683731 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 send_to_waiting 606 -4797> 2013-04-03 10:47:28.683734 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e606 send_incremental [606..606] to client.11041 10.252.24.25:0/3745741276 -4796> 2013-04-03 10:47:28.683754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 build_incremental [606..606] -4795> 2013-04-03 10:47:28.683774 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11041 10.252.24.25:0/3745741276 -- osd_map(606..606 src has 105..606) v3 -- ?+0 0x21fc800 -4794> 2013-04-03 10:47:28.683785 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e606 send_incremental [606..606] to client.11074 10.252.24.32:0/3871608469 -4793> 2013-04-03 10:47:28.683799 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 build_incremental [606..606] -4792> 2013-04-03 10:47:28.683815 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11074 10.252.24.32:0/3871608469 -- osd_map(606..606 src has 105..606) v3 -- ?+0 0x2524800 -4791> 2013-04-03 10:47:28.683826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 committed, telling random osd.2 10.252.24.30:6801/5656 all about it -4790> 2013-04-03 10:47:28.683831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 build_incremental [605..606] -4789> 2013-04-03 10:47:28.683857 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.2 10.252.24.30:6801/5656 -- osd_map(605..606 src has 105..606) v3 -- ?+0 0x2524600 -4788> 2013-04-03 10:47:28.683865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 update_logger -4787> 2013-04-03 10:47:28.683882 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 create_pending e 607 -4786> 2013-04-03 10:47:28.683886 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 remove_redundant_pg_temp -4785> 2013-04-03 10:47:28.683889 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 remove_down_pg_temp -4784> 2013-04-03 10:47:28.683925 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_query osd_boot(osd.3 booted 0 v605) v3 from osd.3 10.252.24.88:6804/18376 -4783> 2013-04-03 10:47:28.683934 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_boot from osd.3 10.252.24.88:6804/18376 -4782> 2013-04-03 10:47:28.683937 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_update osd_boot(osd.3 booted 0 v605) v3 from osd.3 10.252.24.88:6804/18376 -4781> 2013-04-03 10:47:28.683941 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_boot from osd.3 10.252.24.88:6804/18376 sb sb(d6e91ff1-9b43-4972-b7a7-a22d84497d78 osd.3 0b7e2707-d3a7-4f74-93fa-0686f3188d92 e605 [104,605] lci=[576,605]) cluster_addr 10.252.24.88:6805/18376 hb_addr 10.252.24.88:6806/18376 -4780> 2013-04-03 10:47:28.683952 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 setting osd.3 uuid to 0b7e2707-d3a7-4f74-93fa-0686f3188d92 -4779> 2013-04-03 10:47:28.683955 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 old osd_info: up_from 576 up_thru 604 down_at 606 last_clean_interval [541,572) -4778> 2013-04-03 10:47:28.683958 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 prepare_boot osd.3 last_clean_interval [541,572) -> [576-605) -4777> 2013-04-03 10:47:28.683960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 not laggy, new xi down_stamp 2013-04-03 10:47:28.590043 laggy_probability 0 laggy_interval 0 -4776> 2013-04-03 10:47:28.683970 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 should_propose -4775> 2013-04-03 10:47:28.683982 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 update_logger -4774> 2013-04-03 10:47:28.683987 7fcdffcdb700 0 log [INF] : osdmap e606: 5 osds: 4 up, 5 in -4773> 2013-04-03 10:47:28.683988 7fcdffcdb700 10 send_log to self -4772> 2013-04-03 10:47:28.683989 7fcdffcdb700 10 log_queue is 3 last_log 212 sent 211 num 3 unsent 1 sending 1 -4771> 2013-04-03 10:47:28.683990 7fcdffcdb700 10 will send 2013-04-03 10:47:28.638994 mon.0 10.252.24.25:6789/0 211 : [INF] pgmap v29140: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 899B/s wr, 0op/s -4770> 2013-04-03 10:47:28.683995 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2524600 -4769> 2013-04-03 10:47:28.684019 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2524600 con 0x1bac160 -4768> 2013-04-03 10:47:28.684040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos -4767> 2013-04-03 10:47:28.684057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos version 394128 summary v 394128 -4766> 2013-04-03 10:47:28.684068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -4765> 2013-04-03 10:47:28.684078 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 preprocess_log log(1 entries) v1 from mon.0 -4764> 2013-04-03 10:47:28.684096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 nothing new -4763> 2013-04-03 10:47:28.689217 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 19 ==== paxos(lease_ack lc 427560 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (2803235690 0 0) 0x2fe4280 con 0x208a840 -4762> 2013-04-03 10:47:28.739897 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 20 ==== paxos(accept lc 427560 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (2524152190 0 0) 0x3250280 con 0x208a840 -4761> 2013-04-03 10:47:28.741224 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427561 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2fe4280 -4760> 2013-04-03 10:47:28.741245 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427561 fc 427522 pn 0 opn 0) v3 -- ?+0 0x2fe4c80 -4759> 2013-04-03 10:47:28.741341 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29140 update_from_paxos applying incremental 29141 -4758> 2013-04-03 10:47:28.741514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 v29141: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1039B/s wr, 0op/s -4757> 2013-04-03 10:47:28.741545 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 send_pg_creates to 0 pgs -4756> 2013-04-03 10:47:28.741547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 update_logger -4755> 2013-04-03 10:47:28.741563 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 create_pending v 29142 -4754> 2013-04-03 10:47:28.741571 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 check_osd_map already seen 606 >= 606 -4753> 2013-04-03 10:47:28.741573 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 update_logger -4752> 2013-04-03 10:47:28.741583 7fcdffcdb700 0 log [INF] : pgmap v29141: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1039B/s wr, 0op/s -4751> 2013-04-03 10:47:28.741585 7fcdffcdb700 10 send_log to self -4750> 2013-04-03 10:47:28.741585 7fcdffcdb700 10 log_queue is 4 last_log 213 sent 211 num 4 unsent 2 sending 2 -4749> 2013-04-03 10:47:28.741587 7fcdffcdb700 10 will send 2013-04-03 10:47:28.638994 mon.0 10.252.24.25:6789/0 211 : [INF] pgmap v29140: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 899B/s wr, 0op/s -4748> 2013-04-03 10:47:28.741594 7fcdffcdb700 10 will send 2013-04-03 10:47:28.683988 mon.0 10.252.24.25:6789/0 212 : [INF] osdmap e606: 5 osds: 4 up, 5 in -4747> 2013-04-03 10:47:28.741599 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b76600 -4746> 2013-04-03 10:47:28.741643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos -4745> 2013-04-03 10:47:28.741661 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos version 394128 summary v 394128 -4744> 2013-04-03 10:47:28.741679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -4743> 2013-04-03 10:47:28.741700 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b76600 con 0x1bac160 -4742> 2013-04-03 10:47:28.741721 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos -4741> 2013-04-03 10:47:28.741736 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos version 394128 summary v 394128 -4740> 2013-04-03 10:47:28.741745 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4739> 2013-04-03 10:47:28.741756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 preprocess_log log(2 entries) v1 from mon.0 -4738> 2013-04-03 10:47:28.741781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4737> 2013-04-03 10:47:28.741792 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 prepare_log log(2 entries) v1 from mon.0 -4736> 2013-04-03 10:47:28.741802 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 logging 2013-04-03 10:47:28.638994 mon.0 10.252.24.25:6789/0 211 : [INF] pgmap v29140: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 899B/s wr, 0op/s -4735> 2013-04-03 10:47:28.741820 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 logging 2013-04-03 10:47:28.683988 mon.0 10.252.24.25:6789/0 212 : [INF] osdmap e606: 5 osds: 4 up, 5 in -4734> 2013-04-03 10:47:28.773822 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 21 ==== paxos(lease_ack lc 427561 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (620727205 0 0) 0x2fe4c80 con 0x208a840 -4733> 2013-04-03 10:47:28.773870 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos -4732> 2013-04-03 10:47:28.773887 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394128 update_from_paxos version 394128 summary v 394128 -4731> 2013-04-03 10:47:28.773902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -4730> 2013-04-03 10:47:28.773919 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 22 ==== forward(osd_alive(want up_thru 606 have 606) v1) to leader v1 ==== 294+0+0 (3326060920 0 0) 0x2f7ab80 con 0x208a840 -4729> 2013-04-03 10:47:28.773928 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 -4728> 2013-04-03 10:47:28.773937 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x3306c40 from 10.252.24.88:6789/0 -4727> 2013-04-03 10:47:28.773960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_query osd_alive(want up_thru 606 have 606) v1 from osd.4 10.252.24.32:6800/13473 -4726> 2013-04-03 10:47:28.773967 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_alive want up_thru 606 from osd.4 10.252.24.32:6800/13473 -4725> 2013-04-03 10:47:28.773970 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_update osd_alive(want up_thru 606 have 606) v1 from osd.4 10.252.24.32:6800/13473 -4724> 2013-04-03 10:47:28.773973 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_alive want up_thru 606 have 606 from osd.4 10.252.24.32:6800/13473 -4723> 2013-04-03 10:47:28.773977 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 should_propose -4722> 2013-04-03 10:47:28.897861 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.0 10.252.58.31:6802/2682 5 ==== mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=606}) v2 ==== 69+0+0 (281224682 0 0) 0x2e8ba80 con 0x3311580 -4721> 2013-04-03 10:47:28.897891 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=606}) v2 -4720> 2013-04-03 10:47:28.897897 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -4719> 2013-04-03 10:47:28.897902 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e606 send_incremental [606..606] to osd.0 10.252.58.31:6802/2682 -4718> 2013-04-03 10:47:28.898266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 build_incremental [606..606] -4717> 2013-04-03 10:47:28.898294 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.0 10.252.58.31:6802/2682 -- osd_map(606..606 src has 105..606) v3 -- ?+0 0x2b76c00 -4716> 2013-04-03 10:47:28.898308 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.0 10.252.58.31:6802/2682 -- mon_subscribe_ack(300s) v1 -- ?+0 0x3306e00 -4715> 2013-04-03 10:47:28.899524 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.0 10.252.58.31:6802/2682 6 ==== osd_alive(want up_thru 606 have 606) v1 ==== 22+0+0 (2974350322 0 0) 0x336b380 con 0x3311580 -4714> 2013-04-03 10:47:28.899578 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_query osd_alive(want up_thru 606 have 606) v1 from osd.0 10.252.58.31:6802/2682 -4713> 2013-04-03 10:47:28.899585 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_alive want up_thru 606 from osd.0 10.252.58.31:6802/2682 -4712> 2013-04-03 10:47:28.899589 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_update osd_alive(want up_thru 606 have 606) v1 from osd.0 10.252.58.31:6802/2682 -4711> 2013-04-03 10:47:28.899592 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_alive want up_thru 606 have 606 from osd.0 10.252.58.31:6802/2682 -4710> 2013-04-03 10:47:28.899595 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 should_propose -4709> 2013-04-03 10:47:28.947238 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 23 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 210 v187) v2) to leader v1 ==== 560+0+0 (1402062894 0 0) 0x2f7a340 con 0x208a840 -4708> 2013-04-03 10:47:28.947253 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -4707> 2013-04-03 10:47:28.947260 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a51000 from 10.252.24.88:6789/0 -4706> 2013-04-03 10:47:28.947292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e187 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 210 v187) v2 from mds.0 10.252.24.25:6802/3144 -4705> 2013-04-03 10:47:28.947303 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 210 v187) v2 tid 3) v2 -- ?+0 0x24f8240 con 0x208a840 -4704> 2013-04-03 10:47:28.967587 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 24 ==== forward(pg_stats(1 pgs tid 51 v 605) v1) to leader v1 ==== 757+0+0 (1233658120 0 0) 0x2f7ab80 con 0x208a840 -4703> 2013-04-03 10:47:28.967601 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -4702> 2013-04-03 10:47:28.967609 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -4701> 2013-04-03 10:47:28.967643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 preprocess_query pg_stats(1 pgs tid 51 v 605) v1 from osd.2 10.252.24.30:6801/5656 -4700> 2013-04-03 10:47:28.967664 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e606 send_incremental [606..606] to osd.2 10.252.24.30:6801/5656 -4699> 2013-04-03 10:47:28.967679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 build_incremental [606..606] -4698> 2013-04-03 10:47:28.967704 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(606..606 src has 105..606) v3 tid 4) v2 -- ?+0 0x20ad900 con 0x208a840 -4697> 2013-04-03 10:47:28.967713 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 prepare_update pg_stats(1 pgs tid 51 v 605) v1 from osd.2 10.252.24.30:6801/5656 -4696> 2013-04-03 10:47:28.967718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 prepare_pg_stats pg_stats(1 pgs tid 51 v 605) v1 from osd.2 -4695> 2013-04-03 10:47:28.967724 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29141 got osd.2 osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -4694> 2013-04-03 10:47:28.968813 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 25 ==== forward(osd_alive(want up_thru 606 have 606) v1) to leader v1 ==== 294+0+0 (575739676 0 0) 0x2a51840 con 0x208a840 -4693> 2013-04-03 10:47:28.968827 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -4692> 2013-04-03 10:47:28.968834 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x336b700 from 10.252.24.88:6789/0 -4691> 2013-04-03 10:47:28.968865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_query osd_alive(want up_thru 606 have 606) v1 from osd.2 10.252.24.30:6801/5656 -4690> 2013-04-03 10:47:28.968874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 preprocess_alive want up_thru 606 from osd.2 10.252.24.30:6801/5656 -4689> 2013-04-03 10:47:28.968878 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_update osd_alive(want up_thru 606 have 606) v1 from osd.2 10.252.24.30:6801/5656 -4688> 2013-04-03 10:47:28.968880 7fcdffcdb700 7 mon.hotcpc9039@0(leader).osd e606 prepare_alive want up_thru 606 have 606 from osd.2 10.252.24.30:6801/5656 -4687> 2013-04-03 10:47:28.968884 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e606 should_propose -4686> 2013-04-03 10:47:29.107876 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 26 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (424405211 0 0) 0x2f79840 con 0x208a840 -4685> 2013-04-03 10:47:29.107892 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -4684> 2013-04-03 10:47:29.107898 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -4683> 2013-04-03 10:47:29.107905 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 -4682> 2013-04-03 10:47:29.107909 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( start_reply flags( reply-to(mon.2 10.252.24.88:6789/0) ) ) v1 -- ?+0 0x2a51840 -4681> 2013-04-03 10:47:29.431793 7fcdfe7d4700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x29b4a00 sd=25 :6789 s=0 pgs=0 cs=0 l=0).accept sd=25 10.252.24.88:33195/0 -4680> 2013-04-03 10:47:29.432111 7fcdfe7d4700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 ms_verify_authorizer 10.252.24.88:6800/18040 mds protocol 0 -4679> 2013-04-03 10:47:29.432444 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 1 ==== auth(proto 0 36 bytes epoch 0) v1 ==== 66+0+0 (2489530022 0 0) 0x2b76c00 con 0x3311000 -4678> 2013-04-03 10:47:29.432467 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 do not have session, making new one -4677> 2013-04-03 10:47:29.432474 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 ms_dispatch new session MonSession: mds.? 10.252.24.88:6800/18040 is open for mds.? 10.252.24.88:6800/18040 -4676> 2013-04-03 10:47:29.432481 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 setting timeout on session -4675> 2013-04-03 10:47:29.432523 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4674> 2013-04-03 10:47:29.432540 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 preprocess_query auth(proto 0 36 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/18040 -4673> 2013-04-03 10:47:29.432553 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 prep_auth() blob_size=36 -4672> 2013-04-03 10:47:29.432568 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 AuthMonitor::assign_global_id m=auth(proto 0 36 bytes epoch 0) v1 mon=0/3 last_allocated=11901 max_global_id=12096 -4671> 2013-04-03 10:47:29.432579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 next_global_id should be 11904 -4670> 2013-04-03 10:47:29.432596 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mon_map v1 -- ?+0 0x2523800 con 0x3311000 -4669> 2013-04-03 10:47:29.432622 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x20fba00 con 0x3311000 -4668> 2013-04-03 10:47:29.433560 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (130227139 0 0) 0x20fba00 con 0x3311000 -4667> 2013-04-03 10:47:29.433611 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4666> 2013-04-03 10:47:29.433628 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 preprocess_query auth(proto 2 32 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/18040 -4665> 2013-04-03 10:47:29.433639 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 prep_auth() blob_size=32 -4664> 2013-04-03 10:47:29.433755 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x2b76c00 con 0x3311000 -4663> 2013-04-03 10:47:29.434499 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 3 ==== auth(proto 2 181 bytes epoch 0) v1 ==== 211+0+0 (1915732613 0 0) 0x2523800 con 0x3311000 -4662> 2013-04-03 10:47:29.434539 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4661> 2013-04-03 10:47:29.434555 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 preprocess_query auth(proto 2 181 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/18040 -4660> 2013-04-03 10:47:29.434566 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 prep_auth() blob_size=181 -4659> 2013-04-03 10:47:29.434758 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x20fba00 con 0x3311000 -4658> 2013-04-03 10:47:29.435423 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 4 ==== mon_subscribe({monmap=0+}) v2 ==== 23+0+0 (1620593354 0 0) 0x336b8c0 con 0x3311000 -4657> 2013-04-03 10:47:29.435446 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({monmap=0+}) v2 -4656> 2013-04-03 10:47:29.435455 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 0 have 5 -4655> 2013-04-03 10:47:29.435461 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mon_map v1 -- ?+0 0x2523800 con 0x3311000 -4654> 2013-04-03 10:47:29.435466 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/18040 -- mon_subscribe_ack(300s) v1 -- ?+0 0x28a5c40 -4653> 2013-04-03 10:47:29.435475 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 5 ==== auth(proto 2 2 bytes epoch 0) v1 ==== 32+0+0 (2260890001 0 0) 0x287c400 con 0x3311000 -4652> 2013-04-03 10:47:29.435501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4651> 2013-04-03 10:47:29.435518 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 preprocess_query auth(proto 2 2 bytes epoch 0) v1 from mds.? 10.252.24.88:6800/18040 -4650> 2013-04-03 10:47:29.435529 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 prep_auth() blob_size=2 -4649> 2013-04-03 10:47:29.435565 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x2226600 con 0x3311000 -4648> 2013-04-03 10:47:29.436284 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 6 ==== mdsbeacon(11904/hotcpc12533 up:boot seq 1 v0) v2 ==== 289+0+0 (171493784 0 0) 0x273a2c0 con 0x3311000 -4647> 2013-04-03 10:47:29.436331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 preprocess_query mdsbeacon(11904/hotcpc12533 up:boot seq 1 v0) v2 from mds.? 10.252.24.88:6800/18040 -4646> 2013-04-03 10:47:29.436341 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 prepare_update mdsbeacon(11904/hotcpc12533 up:boot seq 1 v0) v2 -4645> 2013-04-03 10:47:29.436345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 fail_mds_gid 11006 mds.hotcpc12533 rank -1 -4644> 2013-04-03 10:47:29.436348 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e606 blacklist 10.252.24.88:6800/17213 until 2013-04-03 11:11:29.436347 -4643> 2013-04-03 10:47:29.436390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e606 min_last_epoch_clean 593 -4642> 2013-04-03 10:47:29.436445 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e606 min_last_epoch_clean 593 -4641> 2013-04-03 10:47:29.436497 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e606 encode_pending e 607 -4640> 2013-04-03 10:47:29.436501 7fcdffcdb700 2 mon.hotcpc9039@0(leader sync( leader state none )).osd e606 osd.3 UP 10.252.24.88:6804/18376 -4639> 2013-04-03 10:47:29.436532 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427561) queue_proposal bl 911 bytes; ctx = 0x1b91470 -4638> 2013-04-03 10:47:29.436541 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427561) propose_queued 427562 911 bytes -4637> 2013-04-03 10:47:29.438042 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427561 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2fe4c80 -4636> 2013-04-03 10:47:29.438062 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 prepare_beacon pending map now: -4635> 2013-04-03 10:47:29.438067 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 print_map epoch 188 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:34:16.921828 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 607 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:active seq 12 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11904: 10.252.24.88:6800/18040 'hotcpc12533' mds.-1.0 up:standby seq 1 -4634> 2013-04-03 10:47:29.438107 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 encode_pending e188 -4633> 2013-04-03 10:47:29.438140 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427522..427561) queue_proposal bl 1079 bytes; ctx = 0x2c3dba0 -4632> 2013-04-03 10:47:29.438147 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427522..427561) propose_new_value not active; proposal queued -4631> 2013-04-03 10:47:29.438153 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 7 ==== mon_subscribe({monmap=6+,osdmap=0}) v2 ==== 42+0+0 (4174948362 0 0) 0x336bc40 con 0x3311000 -4630> 2013-04-03 10:47:29.438162 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({monmap=6+,osdmap=0}) v2 -4629> 2013-04-03 10:47:29.438167 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 -4628> 2013-04-03 10:47:29.438221 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/18040 -- osd_map(606..606 src has 105..606) v3 -- ?+0 0x287c400 -4627> 2013-04-03 10:47:29.438234 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/18040 -- mon_subscribe_ack(300s) v1 -- ?+0 0x336b8c0 -4626> 2013-04-03 10:47:29.438251 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 8 ==== mon_subscribe({mdsmap=0+,monmap=6+,osdmap=0}) v2 ==== 61+0+0 (3602500941 0 0) 0x336ba80 con 0x3311000 -4625> 2013-04-03 10:47:29.438260 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({mdsmap=0+,monmap=6+,osdmap=0}) v2 -4624> 2013-04-03 10:47:29.438273 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/18040 -- mdsmap(e 187) v1 -- ?+0 0x25fb400 -4623> 2013-04-03 10:47:29.438280 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 -4622> 2013-04-03 10:47:29.438318 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/18040 -- osd_map(606..606 src has 105..606) v3 -- ?+0 0x25fb800 -4621> 2013-04-03 10:47:29.438328 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/18040 -- mon_subscribe_ack(300s) v1 -- ?+0 0x336bc40 -4620> 2013-04-03 10:47:29.481852 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 27 ==== paxos(accept lc 427561 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (1793214663 0 0) 0x2fe4c80 con 0x208a840 -4619> 2013-04-03 10:47:29.482179 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427562 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3250280 -4618> 2013-04-03 10:47:29.482206 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427562 fc 427522 pn 0 opn 0) v3 -- ?+0 0x2fe4a00 -4617> 2013-04-03 10:47:29.482277 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e606 update_from_paxos applying incremental 607 -4616> 2013-04-03 10:47:29.482320 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 e607: 5 osds: 5 up, 5 in -4615> 2013-04-03 10:47:29.483430 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 check_osd_map applying osdmap e607 to pg_map -4614> 2013-04-03 10:47:29.483456 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 register_new_pgs checking pg pools for osdmap epoch 607, last_pg_scan 341 -4613> 2013-04-03 10:47:29.483459 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 -4612> 2013-04-03 10:47:29.483463 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -4611> 2013-04-03 10:47:29.483466 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -4610> 2013-04-03 10:47:29.483469 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 -4609> 2013-04-03 10:47:29.483472 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 -4608> 2013-04-03 10:47:29.483475 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 -4607> 2013-04-03 10:47:29.483477 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 -4606> 2013-04-03 10:47:29.483480 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 -4605> 2013-04-03 10:47:29.483483 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 -4604> 2013-04-03 10:47:29.483491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs -4603> 2013-04-03 10:47:29.483494 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 check_down_pgs -4602> 2013-04-03 10:47:29.483551 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 encode_pending v 29142 -4601> 2013-04-03 10:47:29.483586 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427562) queue_proposal bl 654 bytes; ctx = 0x2c3d500 -4600> 2013-04-03 10:47:29.483594 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427562) propose_queued 427563 1079 bytes -4599> 2013-04-03 10:47:29.483819 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427562 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2fe4780 -4598> 2013-04-03 10:47:29.483852 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 send_pg_creates to 0 pgs -4597> 2013-04-03 10:47:29.483862 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_to_waiting 607 -4596> 2013-04-03 10:47:29.483872 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 committed, telling random osd.0 10.252.58.31:6802/2682 all about it -4595> 2013-04-03 10:47:29.483876 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 build_incremental [606..607] -4594> 2013-04-03 10:47:29.483910 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.0 10.252.58.31:6802/2682 -- osd_map(606..607 src has 106..607) v3 -- ?+0 0x2b75a00 -4593> 2013-04-03 10:47:29.483921 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 update_logger -4592> 2013-04-03 10:47:29.483939 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 create_pending e 608 -4591> 2013-04-03 10:47:29.483943 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 remove_redundant_pg_temp -4590> 2013-04-03 10:47:29.483945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 remove_down_pg_temp -4589> 2013-04-03 10:47:29.483963 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 _booted osd.3 10.252.24.88:6804/18376 w 0 from 605 -4588> 2013-04-03 10:47:29.483980 7fcdffcdb700 0 log [INF] : osd.3 10.252.24.88:6804/18376 boot -4587> 2013-04-03 10:47:29.483981 7fcdffcdb700 10 send_log to self -4586> 2013-04-03 10:47:29.483982 7fcdffcdb700 10 log_queue is 5 last_log 214 sent 212 num 5 unsent 2 sending 2 -4585> 2013-04-03 10:47:29.483984 7fcdffcdb700 10 will send 2013-04-03 10:47:28.683988 mon.0 10.252.24.25:6789/0 212 : [INF] osdmap e606: 5 osds: 4 up, 5 in -4584> 2013-04-03 10:47:29.483989 7fcdffcdb700 10 will send 2013-04-03 10:47:28.741584 mon.0 10.252.24.25:6789/0 213 : [INF] pgmap v29141: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1039B/s wr, 0op/s -4583> 2013-04-03 10:47:29.483992 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2523400 -4582> 2013-04-03 10:47:29.484013 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_latest to osd.3 10.252.24.88:6804/18376 start 606 -4581> 2013-04-03 10:47:29.484020 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_incremental [606..607] to osd.3 10.252.24.88:6804/18376 -4580> 2013-04-03 10:47:29.484032 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 build_incremental [606..607] -4579> 2013-04-03 10:47:29.484065 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6804/18376 -- osd_map(606..607 src has 106..607) v3 -- ?+0 0x2169a00 con 0x33112c0 -4578> 2013-04-03 10:47:29.484078 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 _reply_map 606 from osd.4 10.252.24.32:6800/13473 -4577> 2013-04-03 10:47:29.484099 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_latest to osd.4 10.252.24.32:6800/13473 start 606 -4576> 2013-04-03 10:47:29.484105 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_incremental [606..607] to osd.4 10.252.24.32:6800/13473 -4575> 2013-04-03 10:47:29.484116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 build_incremental [606..607] -4574> 2013-04-03 10:47:29.484148 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(606..607 src has 106..607) v3 tid 2) v2 -- ?+0 0x20adb40 con 0x208a840 -4573> 2013-04-03 10:47:29.484158 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 _reply_map 606 from osd.0 10.252.58.31:6802/2682 -4572> 2013-04-03 10:47:29.484182 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_latest to osd.0 10.252.58.31:6802/2682 start 606 -4571> 2013-04-03 10:47:29.484189 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_incremental [606..607] to osd.0 10.252.58.31:6802/2682 -4570> 2013-04-03 10:47:29.484201 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 build_incremental [606..607] -4569> 2013-04-03 10:47:29.484233 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6802/2682 -- osd_map(606..607 src has 106..607) v3 -- ?+0 0x2b76a00 con 0x3311580 -4568> 2013-04-03 10:47:29.484244 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 _reply_map 606 from osd.2 10.252.24.30:6801/5656 -4567> 2013-04-03 10:47:29.484263 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_latest to osd.2 10.252.24.30:6801/5656 start 606 -4566> 2013-04-03 10:47:29.484269 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_incremental [606..607] to osd.2 10.252.24.30:6801/5656 -4565> 2013-04-03 10:47:29.484280 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 build_incremental [606..607] -4564> 2013-04-03 10:47:29.484312 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(606..607 src has 106..607) v3 tid 5) v2 -- ?+0 0x330a000 con 0x208a840 -4563> 2013-04-03 10:47:29.484321 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 update_logger -4562> 2013-04-03 10:47:29.484328 7fcdffcdb700 0 log [INF] : osdmap e607: 5 osds: 5 up, 5 in -4561> 2013-04-03 10:47:29.484329 7fcdffcdb700 10 send_log to self -4560> 2013-04-03 10:47:29.484330 7fcdffcdb700 10 log_queue is 6 last_log 215 sent 213 num 6 unsent 2 sending 2 -4559> 2013-04-03 10:47:29.484331 7fcdffcdb700 10 will send 2013-04-03 10:47:28.741584 mon.0 10.252.24.25:6789/0 213 : [INF] pgmap v29141: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1039B/s wr, 0op/s -4558> 2013-04-03 10:47:29.484341 7fcdffcdb700 10 will send 2013-04-03 10:47:29.483981 mon.0 10.252.24.25:6789/0 214 : [INF] osd.3 10.252.24.88:6804/18376 boot -4557> 2013-04-03 10:47:29.484345 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x24eb200 -4556> 2013-04-03 10:47:29.484371 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2523400 con 0x1bac160 -4555> 2013-04-03 10:47:29.484396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 update_from_paxos -4554> 2013-04-03 10:47:29.484413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 update_from_paxos version 394128 summary v 394128 -4553> 2013-04-03 10:47:29.484422 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4552> 2013-04-03 10:47:29.484434 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 preprocess_log log(2 entries) v1 from mon.0 -4551> 2013-04-03 10:47:29.484465 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4550> 2013-04-03 10:47:29.484478 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 prepare_log log(2 entries) v1 from mon.0 -4549> 2013-04-03 10:47:29.484488 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 logging 2013-04-03 10:47:28.683988 mon.0 10.252.24.25:6789/0 212 : [INF] osdmap e606: 5 osds: 4 up, 5 in -4548> 2013-04-03 10:47:29.484508 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 logging 2013-04-03 10:47:28.741584 mon.0 10.252.24.25:6789/0 213 : [INF] pgmap v29141: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1039B/s wr, 0op/s -4547> 2013-04-03 10:47:29.484540 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x24eb200 con 0x1bac160 -4546> 2013-04-03 10:47:29.484560 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 update_from_paxos -4545> 2013-04-03 10:47:29.484575 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 update_from_paxos version 394128 summary v 394128 -4544> 2013-04-03 10:47:29.484584 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4543> 2013-04-03 10:47:29.484597 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 preprocess_log log(2 entries) v1 from mon.0 -4542> 2013-04-03 10:47:29.484620 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4541> 2013-04-03 10:47:29.484632 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 prepare_log log(2 entries) v1 from mon.0 -4540> 2013-04-03 10:47:29.484844 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 logging 2013-04-03 10:47:28.741584 mon.0 10.252.24.25:6789/0 213 : [INF] pgmap v29141: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1039B/s wr, 0op/s -4539> 2013-04-03 10:47:29.484876 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 logging 2013-04-03 10:47:29.483981 mon.0 10.252.24.25:6789/0 214 : [INF] osd.3 10.252.24.88:6804/18376 boot -4538> 2013-04-03 10:47:29.485932 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.0 10.252.58.31:6802/2682 7 ==== osd_alive(want up_thru 607 have 607) v1 ==== 22+0+0 (806928382 0 0) 0x336b1c0 con 0x3311580 -4537> 2013-04-03 10:47:29.485965 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_query osd_alive(want up_thru 607 have 607) v1 from osd.0 10.252.58.31:6802/2682 -4536> 2013-04-03 10:47:29.485973 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_alive want up_thru 607 from osd.0 10.252.58.31:6802/2682 -4535> 2013-04-03 10:47:29.485977 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_update osd_alive(want up_thru 607 have 607) v1 from osd.0 10.252.58.31:6802/2682 -4534> 2013-04-03 10:47:29.485981 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_alive want up_thru 607 have 607 from osd.0 10.252.58.31:6802/2682 -4533> 2013-04-03 10:47:29.485985 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 should_propose -4532> 2013-04-03 10:47:29.493908 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 28 ==== paxos(lease_ack lc 427562 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3525356102 0 0) 0x2fe4780 con 0x208a840 -4531> 2013-04-03 10:47:29.556397 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 9 ==== osd_alive(want up_thru 607 have 607) v1 ==== 22+0+0 (806928382 0 0) 0x2224700 con 0x33112c0 -4530> 2013-04-03 10:47:29.556443 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_query osd_alive(want up_thru 607 have 607) v1 from osd.3 10.252.24.88:6804/18376 -4529> 2013-04-03 10:47:29.556451 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_alive want up_thru 607 from osd.3 10.252.24.88:6804/18376 -4528> 2013-04-03 10:47:29.556455 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_update osd_alive(want up_thru 607 have 607) v1 from osd.3 10.252.24.88:6804/18376 -4527> 2013-04-03 10:47:29.556459 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_alive want up_thru 607 have 607 from osd.3 10.252.24.88:6804/18376 -4526> 2013-04-03 10:47:29.556462 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 should_propose -4525> 2013-04-03 10:47:29.560931 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 6 ==== mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=606}) v2 ==== 69+0+0 (281224682 0 0) 0x2e8b540 con 0x3311160 -4524> 2013-04-03 10:47:29.560943 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=606}) v2 -4523> 2013-04-03 10:47:29.560949 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 -4522> 2013-04-03 10:47:29.560954 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 send_incremental [606..607] to osd.1 10.252.24.25:6805/3572 -4521> 2013-04-03 10:47:29.560971 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 build_incremental [606..607] -4520> 2013-04-03 10:47:29.560999 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.1 10.252.24.25:6805/3572 -- osd_map(606..607 src has 106..607) v3 -- ?+0 0x287e400 -4519> 2013-04-03 10:47:29.561009 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.1 10.252.24.25:6805/3572 -- mon_subscribe_ack(300s) v1 -- ?+0 0x336b700 -4518> 2013-04-03 10:47:29.561631 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 7 ==== osd_alive(want up_thru 607 have 607) v1 ==== 22+0+0 (806928382 0 0) 0x2224540 con 0x3311160 -4517> 2013-04-03 10:47:29.561699 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_query osd_alive(want up_thru 607 have 607) v1 from osd.1 10.252.24.25:6805/3572 -4516> 2013-04-03 10:47:29.561710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_alive want up_thru 607 from osd.1 10.252.24.25:6805/3572 -4515> 2013-04-03 10:47:29.561714 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_update osd_alive(want up_thru 607 have 607) v1 from osd.1 10.252.24.25:6805/3572 -4514> 2013-04-03 10:47:29.561718 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_alive want up_thru 607 have 607 from osd.1 10.252.24.25:6805/3572 -4513> 2013-04-03 10:47:29.561723 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 should_propose -4512> 2013-04-03 10:47:29.623321 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 29 ==== paxos(accept lc 427562 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (913953511 0 0) 0x3250280 con 0x208a840 -4511> 2013-04-03 10:47:29.675296 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427563 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2fe4780 -4510> 2013-04-03 10:47:29.675320 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427563 fc 427522 pn 0 opn 0) v3 -- ?+0 0x2fe4c80 -4509> 2013-04-03 10:47:29.675387 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 update_from_paxos version 188, my e 187 -4508> 2013-04-03 10:47:29.675405 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e187 update_from_paxos got 188 -4507> 2013-04-03 10:47:29.675419 7fcdffcdb700 4 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 new map -4506> 2013-04-03 10:47:29.675422 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 print_map epoch 188 flags 0 created 2013-01-14 13:01:19.446267 modified 2013-04-03 10:47:29.438111 tableserver 0 root 0 session_timeout 60 session_autoclose 300 last_failure 134 last_failure_osd_epoch 607 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding} max_mds 1 in 0 up {0=11234} failed stopped data_pools 0 metadata_pool 1 11234: 10.252.24.25:6802/3144 'hotcpc9039' mds.0.29 up:active seq 12 10187: 10.252.58.31:6800/10908 'hotcpc12974' mds.-1.0 up:standby seq 1 11904: 10.252.24.88:6800/18040 'hotcpc12533' mds.-1.0 up:standby seq 1 -4505> 2013-04-03 10:47:29.675451 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.58.31:6800/10908 -- mdsmap(e 188) v1 -- ?+0 0x2888600 -4504> 2013-04-03 10:47:29.675464 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11041 10.252.24.25:0/3745741276 -- mdsmap(e 188) v1 -- ?+0 0x2874c00 -4503> 2013-04-03 10:47:29.675477 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11051 10.252.24.30:0/802047285 -- mdsmap(e 188) v1 -- ?+0 0x24e8e00 -4502> 2013-04-03 10:47:29.675489 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11074 10.252.24.32:0/3871608469 -- mdsmap(e 188) v1 -- ?+0 0x2b75400 -4501> 2013-04-03 10:47:29.675511 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11071 10.252.24.32:0/1396772501 -- mdsmap(e 188) v1 -- ?+0 0x209c600 -4500> 2013-04-03 10:47:29.675522 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/18040 -- mdsmap(e 188) v1 -- ?+0 0x2201e00 -4499> 2013-04-03 10:47:29.675530 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 update_logger -4498> 2013-04-03 10:47:29.675548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 create_pending e189 -4497> 2013-04-03 10:47:29.675560 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 _updated mds.? mdsbeacon(11904/hotcpc12533 up:boot seq 1 v0) v2 -4496> 2013-04-03 10:47:29.675570 7fcdffcdb700 0 log [INF] : mds.? 10.252.24.88:6800/18040 up:boot -4495> 2013-04-03 10:47:29.675571 7fcdffcdb700 10 send_log to self -4494> 2013-04-03 10:47:29.675572 7fcdffcdb700 10 log_queue is 7 last_log 216 sent 214 num 7 unsent 2 sending 2 -4493> 2013-04-03 10:47:29.675574 7fcdffcdb700 10 will send 2013-04-03 10:47:29.483981 mon.0 10.252.24.25:6789/0 214 : [INF] osd.3 10.252.24.88:6804/18376 boot -4492> 2013-04-03 10:47:29.675579 7fcdffcdb700 10 will send 2013-04-03 10:47:29.484328 mon.0 10.252.24.25:6789/0 215 : [INF] osdmap e607: 5 osds: 5 up, 5 in -4491> 2013-04-03 10:47:29.675583 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2117400 -4490> 2013-04-03 10:47:29.675602 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -4489> 2013-04-03 10:47:29.675614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 update_logger -4488> 2013-04-03 10:47:29.675620 7fcdffcdb700 0 log [INF] : mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -4487> 2013-04-03 10:47:29.675621 7fcdffcdb700 10 send_log to self -4486> 2013-04-03 10:47:29.675621 7fcdffcdb700 10 log_queue is 8 last_log 217 sent 215 num 8 unsent 2 sending 2 -4485> 2013-04-03 10:47:29.675622 7fcdffcdb700 10 will send 2013-04-03 10:47:29.484328 mon.0 10.252.24.25:6789/0 215 : [INF] osdmap e607: 5 osds: 5 up, 5 in -4484> 2013-04-03 10:47:29.675627 7fcdffcdb700 10 will send 2013-04-03 10:47:29.675571 mon.0 10.252.24.25:6789/0 216 : [INF] mds.? 10.252.24.88:6800/18040 up:boot -4483> 2013-04-03 10:47:29.675630 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2523c00 -4482> 2013-04-03 10:47:29.675654 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427563) propose_queued 427564 654 bytes -4481> 2013-04-03 10:47:29.692715 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427563 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3303000 -4480> 2013-04-03 10:47:29.692737 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2117400 con 0x1bac160 -4479> 2013-04-03 10:47:29.692770 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 update_from_paxos -4478> 2013-04-03 10:47:29.692785 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 update_from_paxos version 394128 summary v 394128 -4477> 2013-04-03 10:47:29.692794 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4476> 2013-04-03 10:47:29.692804 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 preprocess_log log(2 entries) v1 from mon.0 -4475> 2013-04-03 10:47:29.692847 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4474> 2013-04-03 10:47:29.692861 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 prepare_log log(2 entries) v1 from mon.0 -4473> 2013-04-03 10:47:29.692871 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 logging 2013-04-03 10:47:29.483981 mon.0 10.252.24.25:6789/0 214 : [INF] osd.3 10.252.24.88:6804/18376 boot -4472> 2013-04-03 10:47:29.692890 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 logging 2013-04-03 10:47:29.484328 mon.0 10.252.24.25:6789/0 215 : [INF] osdmap e607: 5 osds: 5 up, 5 in -4471> 2013-04-03 10:47:29.692921 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2523c00 con 0x1bac160 -4470> 2013-04-03 10:47:29.692987 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394128 encode_pending v394129 -4469> 2013-04-03 10:47:29.693017 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427522..427563) queue_proposal bl 1652 bytes; ctx = 0x2c3de60 -4468> 2013-04-03 10:47:29.693026 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427522..427563) propose_new_value not active; proposal queued -4467> 2013-04-03 10:47:29.765681 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 30 ==== paxos(lease_ack lc 427563 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (2982556426 0 0) 0x3303000 con 0x208a840 -4466> 2013-04-03 10:47:29.765728 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 31 ==== forward(osd_alive(want up_thru 607 have 607) v1) to leader v1 ==== 294+0+0 (2363254498 0 0) 0x2a83b00 con 0x208a840 -4465> 2013-04-03 10:47:29.765737 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -4464> 2013-04-03 10:47:29.765746 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x336b540 from 10.252.24.88:6789/0 -4463> 2013-04-03 10:47:29.765782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_query osd_alive(want up_thru 607 have 607) v1 from osd.2 10.252.24.30:6801/5656 -4462> 2013-04-03 10:47:29.765790 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 preprocess_alive want up_thru 607 from osd.2 10.252.24.30:6801/5656 -4461> 2013-04-03 10:47:29.765795 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_update osd_alive(want up_thru 607 have 607) v1 from osd.2 10.252.24.30:6801/5656 -4460> 2013-04-03 10:47:29.765800 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 prepare_alive want up_thru 607 have 607 from osd.2 10.252.24.30:6801/5656 -4459> 2013-04-03 10:47:29.765804 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 should_propose -4458> 2013-04-03 10:47:29.777735 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 32 ==== paxos(accept lc 427563 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (340029646 0 0) 0x3303500 con 0x208a840 -4457> 2013-04-03 10:47:29.777992 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427564 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3303000 -4456> 2013-04-03 10:47:29.778023 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427564 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250280 -4455> 2013-04-03 10:47:29.778105 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29141 update_from_paxos applying incremental 29142 -4454> 2013-04-03 10:47:29.778126 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 v29142: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 9167B/s wr, 1op/s -4453> 2013-04-03 10:47:29.778147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 send_pg_creates to 0 pgs -4452> 2013-04-03 10:47:29.778151 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 update_logger -4451> 2013-04-03 10:47:29.778161 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 create_pending v 29143 -4450> 2013-04-03 10:47:29.778170 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 _updated_stats for osd.2 10.252.24.30:6801/5656 -4449> 2013-04-03 10:47:29.778176 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 51) v1 tid 4) v2 -- ?+0 0x20fcb40 con 0x208a840 -4448> 2013-04-03 10:47:29.778187 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 check_osd_map already seen 607 >= 607 -4447> 2013-04-03 10:47:29.778189 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 update_logger -4446> 2013-04-03 10:47:29.778200 7fcdffcdb700 0 log [INF] : pgmap v29142: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 9167B/s wr, 1op/s -4445> 2013-04-03 10:47:29.778201 7fcdffcdb700 10 send_log to self -4444> 2013-04-03 10:47:29.778202 7fcdffcdb700 10 log_queue is 9 last_log 218 sent 216 num 9 unsent 2 sending 2 -4443> 2013-04-03 10:47:29.778204 7fcdffcdb700 10 will send 2013-04-03 10:47:29.675571 mon.0 10.252.24.25:6789/0 216 : [INF] mds.? 10.252.24.88:6800/18040 up:boot -4442> 2013-04-03 10:47:29.778209 7fcdffcdb700 10 will send 2013-04-03 10:47:29.675620 mon.0 10.252.24.25:6789/0 217 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -4441> 2013-04-03 10:47:29.778217 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2201000 -4440> 2013-04-03 10:47:29.778242 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427564) propose_queued 427565 1652 bytes -4439> 2013-04-03 10:47:29.778458 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427564 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2116500 -4438> 2013-04-03 10:47:29.778491 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2201000 con 0x1bac160 -4437> 2013-04-03 10:47:29.802791 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 33 ==== paxos(lease_ack lc 427564 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3797377573 0 0) 0x2116500 con 0x208a840 -4436> 2013-04-03 10:47:29.828069 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 34 ==== paxos(accept lc 427564 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (3043618637 0 0) 0x3250280 con 0x208a840 -4435> 2013-04-03 10:47:29.829282 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427565 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2116500 -4434> 2013-04-03 10:47:29.829317 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427565 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3303500 -4433> 2013-04-03 10:47:29.829355 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos -4432> 2013-04-03 10:47:29.829383 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394128 -4431> 2013-04-03 10:47:29.829398 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos latest full 393798 -4430> 2013-04-03 10:47:29.829416 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos applying incremental log 394129 2013-04-03 10:47:27.483501 mon.0 10.252.24.25:6789/0 210 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -4429> 2013-04-03 10:47:29.829443 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos applying incremental log 394129 2013-04-03 10:47:28.638994 mon.0 10.252.24.25:6789/0 211 : [INF] pgmap v29140: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 899B/s wr, 0op/s -4428> 2013-04-03 10:47:29.829463 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos applying incremental log 394129 2013-04-03 10:47:28.683988 mon.0 10.252.24.25:6789/0 212 : [INF] osdmap e606: 5 osds: 4 up, 5 in -4427> 2013-04-03 10:47:29.829483 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos applying incremental log 394129 2013-04-03 10:47:28.741584 mon.0 10.252.24.25:6789/0 213 : [INF] pgmap v29141: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1039B/s wr, 0op/s -4426> 2013-04-03 10:47:29.829502 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos applying incremental log 394129 2013-04-03 10:47:29.483981 mon.0 10.252.24.25:6789/0 214 : [INF] osd.3 10.252.24.88:6804/18376 boot -4425> 2013-04-03 10:47:29.829521 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos applying incremental log 394129 2013-04-03 10:47:29.484328 mon.0 10.252.24.25:6789/0 215 : [INF] osdmap e607: 5 osds: 5 up, 5 in -4424> 2013-04-03 10:47:29.829566 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 check_subs -4423> 2013-04-03 10:47:29.829601 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 create_pending v 394130 -4422> 2013-04-03 10:47:29.829626 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 _updated_log for mon.0 10.252.24.25:6789/0 -4421> 2013-04-03 10:47:29.829639 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 211) v1 -- ?+0 0x2e8b540 con 0x1bac160 -4420> 2013-04-03 10:47:29.829645 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 _updated_log for mon.0 10.252.24.25:6789/0 -4419> 2013-04-03 10:47:29.829661 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 212) v1 -- ?+0 0x336be00 con 0x1bac160 -4418> 2013-04-03 10:47:29.829666 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 _updated_log for mon.0 10.252.24.25:6789/0 -4417> 2013-04-03 10:47:29.829679 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 213) v1 -- ?+0 0x336b8c0 con 0x1bac160 -4416> 2013-04-03 10:47:29.829683 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 _updated_log for mon.0 10.252.24.25:6789/0 -4415> 2013-04-03 10:47:29.829696 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 214) v1 -- ?+0 0x336b380 con 0x1bac160 -4414> 2013-04-03 10:47:29.829700 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 _updated_log for mon.0 10.252.24.25:6789/0 -4413> 2013-04-03 10:47:29.829713 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 215) v1 -- ?+0 0x336ba80 con 0x1bac160 -4412> 2013-04-03 10:47:29.829730 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos -4411> 2013-04-03 10:47:29.830094 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 -4410> 2013-04-03 10:47:29.830108 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4409> 2013-04-03 10:47:29.830121 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_log log(2 entries) v1 from mon.0 -4408> 2013-04-03 10:47:29.830147 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4407> 2013-04-03 10:47:29.830159 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_log log(2 entries) v1 from mon.0 -4406> 2013-04-03 10:47:29.830170 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:29.484328 mon.0 10.252.24.25:6789/0 215 : [INF] osdmap e607: 5 osds: 5 up, 5 in -4405> 2013-04-03 10:47:29.830189 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:29.675571 mon.0 10.252.24.25:6789/0 216 : [INF] mds.? 10.252.24.88:6800/18040 up:boot -4404> 2013-04-03 10:47:29.830232 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos -4403> 2013-04-03 10:47:29.830247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 -4402> 2013-04-03 10:47:29.830256 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4401> 2013-04-03 10:47:29.830267 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_log log(2 entries) v1 from mon.0 -4400> 2013-04-03 10:47:29.830290 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4399> 2013-04-03 10:47:29.830303 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_log log(2 entries) v1 from mon.0 -4398> 2013-04-03 10:47:29.830313 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:29.675571 mon.0 10.252.24.25:6789/0 216 : [INF] mds.? 10.252.24.88:6800/18040 up:boot -4397> 2013-04-03 10:47:29.830331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:29.675620 mon.0 10.252.24.25:6789/0 217 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -4396> 2013-04-03 10:47:29.830396 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos -4395> 2013-04-03 10:47:29.830412 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 -4394> 2013-04-03 10:47:29.830431 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4393> 2013-04-03 10:47:29.830452 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 211) v1 ==== 0+0+0 (0 0 0) 0x2e8b540 con 0x1bac160 -4392> 2013-04-03 10:47:29.830460 7fcdffcdb700 10 handle_log_ack log(last 211) v1 -4391> 2013-04-03 10:47:29.830462 7fcdffcdb700 10 logged 2013-04-03 10:47:27.483501 mon.0 10.252.24.25:6789/0 210 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -4390> 2013-04-03 10:47:29.830467 7fcdffcdb700 10 logged 2013-04-03 10:47:28.638994 mon.0 10.252.24.25:6789/0 211 : [INF] pgmap v29140: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 899B/s wr, 0op/s -4389> 2013-04-03 10:47:29.830473 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 212) v1 ==== 0+0+0 (0 0 0) 0x336be00 con 0x1bac160 -4388> 2013-04-03 10:47:29.830478 7fcdffcdb700 10 handle_log_ack log(last 212) v1 -4387> 2013-04-03 10:47:29.830479 7fcdffcdb700 10 logged 2013-04-03 10:47:28.683988 mon.0 10.252.24.25:6789/0 212 : [INF] osdmap e606: 5 osds: 4 up, 5 in -4386> 2013-04-03 10:47:29.830484 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 213) v1 ==== 0+0+0 (0 0 0) 0x336b8c0 con 0x1bac160 -4385> 2013-04-03 10:47:29.830490 7fcdffcdb700 10 handle_log_ack log(last 213) v1 -4384> 2013-04-03 10:47:29.830493 7fcdffcdb700 10 logged 2013-04-03 10:47:28.741584 mon.0 10.252.24.25:6789/0 213 : [INF] pgmap v29141: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1039B/s wr, 0op/s -4383> 2013-04-03 10:47:29.830499 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 214) v1 ==== 0+0+0 (0 0 0) 0x336b380 con 0x1bac160 -4382> 2013-04-03 10:47:29.830504 7fcdffcdb700 10 handle_log_ack log(last 214) v1 -4381> 2013-04-03 10:47:29.830505 7fcdffcdb700 10 logged 2013-04-03 10:47:29.483981 mon.0 10.252.24.25:6789/0 214 : [INF] osd.3 10.252.24.88:6804/18376 boot -4380> 2013-04-03 10:47:29.830510 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 215) v1 ==== 0+0+0 (0 0 0) 0x336ba80 con 0x1bac160 -4379> 2013-04-03 10:47:29.830516 7fcdffcdb700 10 handle_log_ack log(last 215) v1 -4378> 2013-04-03 10:47:29.830517 7fcdffcdb700 10 logged 2013-04-03 10:47:29.484328 mon.0 10.252.24.25:6789/0 215 : [INF] osdmap e607: 5 osds: 5 up, 5 in -4377> 2013-04-03 10:47:29.853416 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 35 ==== paxos(lease_ack lc 427565 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1458617352 0 0) 0x3303500 con 0x208a840 -4376> 2013-04-03 10:47:29.853461 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos -4375> 2013-04-03 10:47:29.853479 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 -4374> 2013-04-03 10:47:29.853494 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4373> 2013-04-03 10:47:30.482025 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 min_last_epoch_clean 593 -4372> 2013-04-03 10:47:30.482093 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 min_last_epoch_clean 593 -4371> 2013-04-03 10:47:30.482148 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 encode_pending e 608 -4370> 2013-04-03 10:47:30.482183 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427565) queue_proposal bl 317 bytes; ctx = 0x2c3df40 -4369> 2013-04-03 10:47:30.482196 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427565) propose_queued 427566 317 bytes -4368> 2013-04-03 10:47:30.483890 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427565 fc 0 pn 17700 opn 0) v3 -- ?+0 0x300ca00 -4367> 2013-04-03 10:47:30.511815 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 36 ==== paxos(accept lc 427565 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (300437988 0 0) 0x300ca00 con 0x208a840 -4366> 2013-04-03 10:47:30.512124 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427566 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3303500 -4365> 2013-04-03 10:47:30.512152 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427566 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250280 -4364> 2013-04-03 10:47:30.512222 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e607 update_from_paxos applying incremental 608 -4363> 2013-04-03 10:47:30.512267 7fcdffcdb700 1 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in -4362> 2013-04-03 10:47:30.513383 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 check_osd_map applying osdmap e608 to pg_map -4361> 2013-04-03 10:47:30.513407 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 register_new_pgs checking pg pools for osdmap epoch 608, last_pg_scan 341 -4360> 2013-04-03 10:47:30.513412 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 0 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 -4359> 2013-04-03 10:47:30.513422 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 1 rep size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -4358> 2013-04-03 10:47:30.513425 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 2 rep size 2 min_size 1 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 -4357> 2013-04-03 10:47:30.513428 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 3 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 301 owner 18446744073709551615 -4356> 2013-04-03 10:47:30.513431 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 4 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 302 owner 18446744073709551615 -4355> 2013-04-03 10:47:30.513433 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 5 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 304 owner 18446744073709551615 -4354> 2013-04-03 10:47:30.513436 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 6 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 321 owner 18446744073709551615 -4353> 2013-04-03 10:47:30.513439 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 7 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 323 owner 18446744073709551615 -4352> 2013-04-03 10:47:30.513444 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 no change in pool 8 rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 341 owner 18446744073709551615 -4351> 2013-04-03 10:47:30.513453 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 register_new_pgs registered 0 new pgs, removed 0 uncreated pgs -4350> 2013-04-03 10:47:30.513456 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 check_down_pgs -4349> 2013-04-03 10:47:30.513514 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 encode_pending v 29143 -4348> 2013-04-03 10:47:30.513543 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427566) queue_proposal bl 215 bytes; ctx = 0x2c3d910 -4347> 2013-04-03 10:47:30.513551 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427566) propose_queued 427567 215 bytes -4346> 2013-04-03 10:47:30.513761 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427566 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2116780 -4345> 2013-04-03 10:47:30.513777 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 send_pg_creates to 0 pgs -4344> 2013-04-03 10:47:30.513782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_to_waiting 608 -4343> 2013-04-03 10:47:30.513786 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 committed, telling random osd.1 10.252.24.25:6805/3572 all about it -4342> 2013-04-03 10:47:30.513790 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 build_incremental [607..608] -4341> 2013-04-03 10:47:30.513824 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.1 10.252.24.25:6805/3572 -- osd_map(607..608 src has 107..608) v3 -- ?+0 0x2117400 -4340> 2013-04-03 10:47:30.513837 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 update_logger -4339> 2013-04-03 10:47:30.513856 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 create_pending e 609 -4338> 2013-04-03 10:47:30.513861 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 remove_redundant_pg_temp -4337> 2013-04-03 10:47:30.513863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 remove_down_pg_temp -4336> 2013-04-03 10:47:30.513882 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 _reply_map 607 from osd.0 10.252.58.31:6802/2682 -4335> 2013-04-03 10:47:30.513899 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_latest to osd.0 10.252.58.31:6802/2682 start 607 -4334> 2013-04-03 10:47:30.513906 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_incremental [607..608] to osd.0 10.252.58.31:6802/2682 -4333> 2013-04-03 10:47:30.513918 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 build_incremental [607..608] -4332> 2013-04-03 10:47:30.513951 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6802/2682 -- osd_map(607..608 src has 107..608) v3 -- ?+0 0x24eb200 con 0x3311580 -4331> 2013-04-03 10:47:30.513962 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 _reply_map 607 from osd.3 10.252.24.88:6804/18376 -4330> 2013-04-03 10:47:30.513982 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_latest to osd.3 10.252.24.88:6804/18376 start 607 -4329> 2013-04-03 10:47:30.513988 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_incremental [607..608] to osd.3 10.252.24.88:6804/18376 -4328> 2013-04-03 10:47:30.513999 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 build_incremental [607..608] -4327> 2013-04-03 10:47:30.514031 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6804/18376 -- osd_map(607..608 src has 107..608) v3 -- ?+0 0x2523400 con 0x33112c0 -4326> 2013-04-03 10:47:30.514040 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 _reply_map 607 from osd.1 10.252.24.25:6805/3572 -4325> 2013-04-03 10:47:30.514058 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_latest to osd.1 10.252.24.25:6805/3572 start 607 -4324> 2013-04-03 10:47:30.514064 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_incremental [607..608] to osd.1 10.252.24.25:6805/3572 -4323> 2013-04-03 10:47:30.514075 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 build_incremental [607..608] -4322> 2013-04-03 10:47:30.514107 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- osd_map(607..608 src has 107..608) v3 -- ?+0 0x2b76600 con 0x3311160 -4321> 2013-04-03 10:47:30.514117 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 _reply_map 607 from osd.2 10.252.24.30:6801/5656 -4320> 2013-04-03 10:47:30.514135 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_latest to osd.2 10.252.24.30:6801/5656 start 607 -4319> 2013-04-03 10:47:30.514141 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_incremental [607..608] to osd.2 10.252.24.30:6801/5656 -4318> 2013-04-03 10:47:30.514153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 build_incremental [607..608] -4317> 2013-04-03 10:47:30.514184 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(osd_map(607..608 src has 107..608) v3 tid 6) v2 -- ?+0 0x24f8240 con 0x208a840 -4316> 2013-04-03 10:47:30.514199 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 update_logger -4315> 2013-04-03 10:47:30.514206 7fcdffcdb700 0 log [INF] : osdmap e608: 5 osds: 5 up, 5 in -4314> 2013-04-03 10:47:30.514207 7fcdffcdb700 10 send_log to self -4313> 2013-04-03 10:47:30.514208 7fcdffcdb700 10 log_queue is 4 last_log 219 sent 217 num 4 unsent 2 sending 2 -4312> 2013-04-03 10:47:30.514210 7fcdffcdb700 10 will send 2013-04-03 10:47:29.675620 mon.0 10.252.24.25:6789/0 217 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -4311> 2013-04-03 10:47:30.514215 7fcdffcdb700 10 will send 2013-04-03 10:47:29.778201 mon.0 10.252.24.25:6789/0 218 : [INF] pgmap v29142: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 9167B/s wr, 1op/s -4310> 2013-04-03 10:47:30.514219 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x24eb600 -4309> 2013-04-03 10:47:30.514245 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x24eb600 con 0x1bac160 -4308> 2013-04-03 10:47:30.514267 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos -4307> 2013-04-03 10:47:30.514282 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 -4306> 2013-04-03 10:47:30.514291 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4305> 2013-04-03 10:47:30.514301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_log log(2 entries) v1 from mon.0 -4304> 2013-04-03 10:47:30.514331 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4303> 2013-04-03 10:47:30.514342 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_log log(2 entries) v1 from mon.0 -4302> 2013-04-03 10:47:30.514353 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:29.675620 mon.0 10.252.24.25:6789/0 217 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -4301> 2013-04-03 10:47:30.514371 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:29.778201 mon.0 10.252.24.25:6789/0 218 : [INF] pgmap v29142: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 9167B/s wr, 1op/s -4300> 2013-04-03 10:47:30.515059 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 10 ==== mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=608}) v2 ==== 69+0+0 (83000142 0 0) 0x2e8b000 con 0x33112c0 -4299> 2013-04-03 10:47:30.515073 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=608}) v2 -4298> 2013-04-03 10:47:30.515079 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 check_sub monmap next 6 have 5 -4297> 2013-04-03 10:47:30.515085 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 send_incremental [608..608] to osd.3 10.252.24.88:6804/18376 -4296> 2013-04-03 10:47:30.515116 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 build_incremental [608..608] -4295> 2013-04-03 10:47:30.515137 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.3 10.252.24.88:6804/18376 -- osd_map(608..608 src has 107..608) v3 -- ?+0 0x2b78400 -4294> 2013-04-03 10:47:30.515147 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.3 10.252.24.88:6804/18376 -- mon_subscribe_ack(300s) v1 -- ?+0 0x336b540 -4293> 2013-04-03 10:47:30.523822 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 37 ==== paxos(lease_ack lc 427566 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3638919445 0 0) 0x2116780 con 0x208a840 -4292> 2013-04-03 10:47:30.667687 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 38 ==== paxos(accept lc 427566 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (233098671 0 0) 0x3250280 con 0x208a840 -4291> 2013-04-03 10:47:30.668861 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427567 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2116780 -4290> 2013-04-03 10:47:30.668885 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427567 fc 427522 pn 0 opn 0) v3 -- ?+0 0x300ca00 -4289> 2013-04-03 10:47:30.668963 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29142 update_from_paxos applying incremental 29143 -4288> 2013-04-03 10:47:30.668973 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 v29143: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 4466B/s wr, 0op/s -4287> 2013-04-03 10:47:30.668993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 send_pg_creates to 0 pgs -4286> 2013-04-03 10:47:30.668996 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 update_logger -4285> 2013-04-03 10:47:30.669006 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 create_pending v 29144 -4284> 2013-04-03 10:47:30.669017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 check_osd_map already seen 608 >= 608 -4283> 2013-04-03 10:47:30.669021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 update_logger -4282> 2013-04-03 10:47:30.669032 7fcdffcdb700 0 log [INF] : pgmap v29143: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 4466B/s wr, 0op/s -4281> 2013-04-03 10:47:30.669033 7fcdffcdb700 10 send_log to self -4280> 2013-04-03 10:47:30.669034 7fcdffcdb700 10 log_queue is 5 last_log 220 sent 218 num 5 unsent 2 sending 2 -4279> 2013-04-03 10:47:30.669036 7fcdffcdb700 10 will send 2013-04-03 10:47:29.778201 mon.0 10.252.24.25:6789/0 218 : [INF] pgmap v29142: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 9167B/s wr, 1op/s -4278> 2013-04-03 10:47:30.669043 7fcdffcdb700 10 will send 2013-04-03 10:47:30.514207 mon.0 10.252.24.25:6789/0 219 : [INF] osdmap e608: 5 osds: 5 up, 5 in -4277> 2013-04-03 10:47:30.669047 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2524c00 -4276> 2013-04-03 10:47:30.669097 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos -4275> 2013-04-03 10:47:30.669118 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 -4274> 2013-04-03 10:47:30.669137 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4273> 2013-04-03 10:47:30.669163 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2524c00 con 0x1bac160 -4272> 2013-04-03 10:47:30.669186 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos -4271> 2013-04-03 10:47:30.669202 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 -4270> 2013-04-03 10:47:30.669212 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4269> 2013-04-03 10:47:30.669223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 preprocess_log log(2 entries) v1 from mon.0 -4268> 2013-04-03 10:47:30.669253 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4267> 2013-04-03 10:47:30.669265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 prepare_log log(2 entries) v1 from mon.0 -4266> 2013-04-03 10:47:30.669276 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:29.778201 mon.0 10.252.24.25:6789/0 218 : [INF] pgmap v29142: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 9167B/s wr, 1op/s -4265> 2013-04-03 10:47:30.669297 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 logging 2013-04-03 10:47:30.514207 mon.0 10.252.24.25:6789/0 219 : [INF] osdmap e608: 5 osds: 5 up, 5 in -4264> 2013-04-03 10:47:30.725537 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 39 ==== paxos(lease_ack lc 427567 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (341252004 0 0) 0x300ca00 con 0x208a840 -4263> 2013-04-03 10:47:30.725605 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos -4262> 2013-04-03 10:47:30.725625 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 update_from_paxos version 394129 summary v 394129 -4261> 2013-04-03 10:47:30.725643 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4260> 2013-04-03 10:47:30.828265 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394129 encode_pending v394130 -4259> 2013-04-03 10:47:30.828301 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427567) queue_proposal bl 1113 bytes; ctx = 0x2c3d5c0 -4258> 2013-04-03 10:47:30.828309 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427567) propose_queued 427568 1113 bytes -4257> 2013-04-03 10:47:30.883021 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427567 fc 0 pn 17700 opn 0) v3 -- ?+0 0x300c780 -4256> 2013-04-03 10:47:30.958543 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 40 ==== paxos(accept lc 427567 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (3727435934 0 0) 0x300c780 con 0x208a840 -4255> 2013-04-03 10:47:30.970088 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427568 fc 0 pn 17700 opn 0) v3 -- ?+0 0x300ca00 -4254> 2013-04-03 10:47:30.970107 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427568 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250280 -4253> 2013-04-03 10:47:30.970149 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos -4252> 2013-04-03 10:47:30.970177 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos version 394130 summary v 394129 -4251> 2013-04-03 10:47:30.970192 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos latest full 393798 -4250> 2013-04-03 10:47:30.970210 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos applying incremental log 394130 2013-04-03 10:47:29.675571 mon.0 10.252.24.25:6789/0 216 : [INF] mds.? 10.252.24.88:6800/18040 up:boot -4249> 2013-04-03 10:47:30.970239 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos applying incremental log 394130 2013-04-03 10:47:29.675620 mon.0 10.252.24.25:6789/0 217 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -4248> 2013-04-03 10:47:30.970260 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos applying incremental log 394130 2013-04-03 10:47:29.778201 mon.0 10.252.24.25:6789/0 218 : [INF] pgmap v29142: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 9167B/s wr, 1op/s -4247> 2013-04-03 10:47:30.970280 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos applying incremental log 394130 2013-04-03 10:47:30.514207 mon.0 10.252.24.25:6789/0 219 : [INF] osdmap e608: 5 osds: 5 up, 5 in -4246> 2013-04-03 10:47:30.970314 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 check_subs -4245> 2013-04-03 10:47:30.970349 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 create_pending v 394131 -4244> 2013-04-03 10:47:30.970372 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 _updated_log for mon.0 10.252.24.25:6789/0 -4243> 2013-04-03 10:47:30.970386 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 216) v1 -- ?+0 0x2e8b000 con 0x1bac160 -4242> 2013-04-03 10:47:30.970393 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 _updated_log for mon.0 10.252.24.25:6789/0 -4241> 2013-04-03 10:47:30.970410 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 217) v1 -- ?+0 0x336b1c0 con 0x1bac160 -4240> 2013-04-03 10:47:30.970415 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 _updated_log for mon.0 10.252.24.25:6789/0 -4239> 2013-04-03 10:47:30.970428 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 218) v1 -- ?+0 0x336ba80 con 0x1bac160 -4238> 2013-04-03 10:47:30.970433 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 _updated_log for mon.0 10.252.24.25:6789/0 -4237> 2013-04-03 10:47:30.970446 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 219) v1 -- ?+0 0x336b380 con 0x1bac160 -4236> 2013-04-03 10:47:30.970495 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos -4235> 2013-04-03 10:47:30.970511 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos version 394130 summary v 394130 -4234> 2013-04-03 10:47:30.970529 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4233> 2013-04-03 10:47:30.970550 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 216) v1 ==== 0+0+0 (0 0 0) 0x2e8b000 con 0x1bac160 -4232> 2013-04-03 10:47:30.970557 7fcdffcdb700 10 handle_log_ack log(last 216) v1 -4231> 2013-04-03 10:47:30.970559 7fcdffcdb700 10 logged 2013-04-03 10:47:29.675571 mon.0 10.252.24.25:6789/0 216 : [INF] mds.? 10.252.24.88:6800/18040 up:boot -4230> 2013-04-03 10:47:30.970565 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 217) v1 ==== 0+0+0 (0 0 0) 0x336b1c0 con 0x1bac160 -4229> 2013-04-03 10:47:30.970570 7fcdffcdb700 10 handle_log_ack log(last 217) v1 -4228> 2013-04-03 10:47:30.970571 7fcdffcdb700 10 logged 2013-04-03 10:47:29.675620 mon.0 10.252.24.25:6789/0 217 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -4227> 2013-04-03 10:47:30.970576 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 218) v1 ==== 0+0+0 (0 0 0) 0x336ba80 con 0x1bac160 -4226> 2013-04-03 10:47:30.970582 7fcdffcdb700 10 handle_log_ack log(last 218) v1 -4225> 2013-04-03 10:47:30.970583 7fcdffcdb700 10 logged 2013-04-03 10:47:29.778201 mon.0 10.252.24.25:6789/0 218 : [INF] pgmap v29142: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 9167B/s wr, 1op/s -4224> 2013-04-03 10:47:30.970588 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 219) v1 ==== 0+0+0 (0 0 0) 0x336b380 con 0x1bac160 -4223> 2013-04-03 10:47:30.970593 7fcdffcdb700 10 handle_log_ack log(last 219) v1 -4222> 2013-04-03 10:47:30.970596 7fcdffcdb700 10 logged 2013-04-03 10:47:30.514207 mon.0 10.252.24.25:6789/0 219 : [INF] osdmap e608: 5 osds: 5 up, 5 in -4221> 2013-04-03 10:47:30.995851 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 41 ==== paxos(lease_ack lc 427568 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (585638707 0 0) 0x3250280 con 0x208a840 -4220> 2013-04-03 10:47:30.995896 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos -4219> 2013-04-03 10:47:30.995914 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos version 394130 summary v 394130 -4218> 2013-04-03 10:47:30.995929 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4217> 2013-04-03 10:47:31.600796 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 213 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140496 v188) v2 ==== 289+0+0 (3179879564 0 0) 0x20e3dc0 con 0x2118420 -4216> 2013-04-03 10:47:31.600863 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140496 v188) v2 from mds.? 10.252.58.31:6800/10908 -4215> 2013-04-03 10:47:31.600876 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140496 v188) v2 -- ?+0 0x2a83b00 con 0x2118420 -4214> 2013-04-03 10:47:32.947393 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 42 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 211 v188) v2) to leader v1 ==== 560+0+0 (3720517888 0 0) 0x2a84340 con 0x208a840 -4213> 2013-04-03 10:47:32.947414 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -4212> 2013-04-03 10:47:32.947427 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x273b600 from 10.252.24.88:6789/0 -4211> 2013-04-03 10:47:32.947475 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 211 v188) v2 from mds.0 10.252.24.25:6802/3144 -4210> 2013-04-03 10:47:32.947488 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 211 v188) v2 tid 7) v2 -- ?+0 0x24f8d80 con 0x208a840 -4209> 2013-04-03 10:47:33.385343 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 check_down_pgs -4208> 2013-04-03 10:47:33.385374 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 v29143: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 4466B/s wr, 0op/s -4207> 2013-04-03 10:47:33.385410 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -4206> 2013-04-03 10:47:33.385434 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in -4205> 2013-04-03 10:47:33.385472 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 593 -4204> 2013-04-03 10:47:33.385505 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos -4203> 2013-04-03 10:47:33.385524 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos version 394130 summary v 394130 -4202> 2013-04-03 10:47:33.385534 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 log -4201> 2013-04-03 10:47:33.385551 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4200> 2013-04-03 10:47:33.385569 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth -4199> 2013-04-03 10:47:33.436415 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 9 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 2 v188) v2 ==== 289+0+0 (158072658 0 0) 0x2a83000 con 0x3311000 -4198> 2013-04-03 10:47:33.436462 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 2 v188) v2 from mds.? 10.252.24.88:6800/18040 -4197> 2013-04-03 10:47:33.436474 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 2 v188) v2 -- ?+0 0x2a84340 con 0x3311000 -4196> 2013-04-03 10:47:33.848696 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 11 ==== pg_stats(73 pgs tid 1 v 608) v1 ==== 27341+0+0 (1065532715 0 0) 0x2b26d80 con 0x33112c0 -4195> 2013-04-03 10:47:33.848769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 preprocess_query pg_stats(73 pgs tid 1 v 608) v1 from osd.3 10.252.24.88:6804/18376 -4194> 2013-04-03 10:47:33.848779 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 prepare_update pg_stats(73 pgs tid 1 v 608) v1 from osd.3 10.252.24.88:6804/18376 -4193> 2013-04-03 10:47:33.848789 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 prepare_pg_stats pg_stats(73 pgs tid 1 v 608) v1 from osd.3 -4192> 2013-04-03 10:47:33.848797 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 got osd.3 osd_stat(1188 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -4191> 2013-04-03 10:47:33.899491 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 encode_pending v 29144 -4190> 2013-04-03 10:47:33.899704 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427568) queue_proposal bl 27510 bytes; ctx = 0x2c3df70 -4189> 2013-04-03 10:47:33.899715 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427568) propose_queued 427569 27510 bytes -4188> 2013-04-03 10:47:33.901237 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427568 fc 0 pn 17700 opn 0) v3 -- ?+0 0x300cc80 -4187> 2013-04-03 10:47:33.932877 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 43 ==== paxos(accept lc 427568 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (3771764005 0 0) 0x300cc80 con 0x208a840 -4186> 2013-04-03 10:47:33.933922 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427569 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3250280 -4185> 2013-04-03 10:47:33.933941 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427569 fc 427522 pn 0 opn 0) v3 -- ?+0 0x300c780 -4184> 2013-04-03 10:47:33.934035 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29143 update_from_paxos applying incremental 29144 -4183> 2013-04-03 10:47:33.934218 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 v29144: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail -4182> 2013-04-03 10:47:33.934247 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 send_pg_creates to 0 pgs -4181> 2013-04-03 10:47:33.934252 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 update_logger -4180> 2013-04-03 10:47:33.934271 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 create_pending v 29145 -4179> 2013-04-03 10:47:33.934281 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 _updated_stats for osd.3 10.252.24.88:6804/18376 -4178> 2013-04-03 10:47:33.934288 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6804/18376 -- pg_stats_ack(73 pgs tid 1) v1 -- ?+0 0x336b380 con 0x33112c0 -4177> 2013-04-03 10:47:33.934311 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 check_osd_map already seen 608 >= 608 -4176> 2013-04-03 10:47:33.934315 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 update_logger -4175> 2013-04-03 10:47:33.934325 7fcdffcdb700 0 log [INF] : pgmap v29144: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail -4174> 2013-04-03 10:47:33.934327 7fcdffcdb700 10 send_log to self -4173> 2013-04-03 10:47:33.934328 7fcdffcdb700 10 log_queue is 2 last_log 221 sent 219 num 2 unsent 2 sending 2 -4172> 2013-04-03 10:47:33.934329 7fcdffcdb700 10 will send 2013-04-03 10:47:30.669033 mon.0 10.252.24.25:6789/0 220 : [INF] pgmap v29143: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 4466B/s wr, 0op/s -4171> 2013-04-03 10:47:33.934335 7fcdffcdb700 10 will send 2013-04-03 10:47:33.934326 mon.0 10.252.24.25:6789/0 221 : [INF] pgmap v29144: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail -4170> 2013-04-03 10:47:33.934341 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2b78000 -4169> 2013-04-03 10:47:33.934391 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos -4168> 2013-04-03 10:47:33.934412 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos version 394130 summary v 394130 -4167> 2013-04-03 10:47:33.934431 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4166> 2013-04-03 10:47:33.934454 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2b78000 con 0x1bac160 -4165> 2013-04-03 10:47:33.934476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos -4164> 2013-04-03 10:47:33.934491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos version 394130 summary v 394130 -4163> 2013-04-03 10:47:33.934501 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4162> 2013-04-03 10:47:33.934511 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 preprocess_log log(2 entries) v1 from mon.0 -4161> 2013-04-03 10:47:33.934542 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -4160> 2013-04-03 10:47:33.934554 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 prepare_log log(2 entries) v1 from mon.0 -4159> 2013-04-03 10:47:33.934564 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 logging 2013-04-03 10:47:30.669033 mon.0 10.252.24.25:6789/0 220 : [INF] pgmap v29143: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 4466B/s wr, 0op/s -4158> 2013-04-03 10:47:33.934585 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 logging 2013-04-03 10:47:33.934326 mon.0 10.252.24.25:6789/0 221 : [INF] pgmap v29144: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail -4157> 2013-04-03 10:47:33.966827 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 44 ==== paxos(lease_ack lc 427569 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3029799258 0 0) 0x300c780 con 0x208a840 -4156> 2013-04-03 10:47:33.966871 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos -4155> 2013-04-03 10:47:33.966889 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 update_from_paxos version 394130 summary v 394130 -4154> 2013-04-03 10:47:33.966904 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4153> 2013-04-03 10:47:34.551206 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 45 ==== forward(pg_stats(0 pgs tid 64 v 608) v1) to leader v1 ==== 368+0+0 (1977854468 0 0) 0x20e3dc0 con 0x208a840 -4152> 2013-04-03 10:47:34.551227 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 -4151> 2013-04-03 10:47:34.551240 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8d80 from 10.252.24.88:6789/0 -4150> 2013-04-03 10:47:34.551292 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 preprocess_query pg_stats(0 pgs tid 64 v 608) v1 from osd.4 10.252.24.32:6800/13473 -4149> 2013-04-03 10:47:34.551301 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 prepare_update pg_stats(0 pgs tid 64 v 608) v1 from osd.4 10.252.24.32:6800/13473 -4148> 2013-04-03 10:47:34.551305 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 prepare_pg_stats pg_stats(0 pgs tid 64 v 608) v1 from osd.4 -4147> 2013-04-03 10:47:34.551312 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 got osd.4 osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[])) -4146> 2013-04-03 10:47:34.644392 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 8 ==== pg_stats(45 pgs tid 70 v 608) v1 ==== 16889+0+0 (170961493 0 0) 0x2b26b40 con 0x3311160 -4145> 2013-04-03 10:47:34.644462 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 preprocess_query pg_stats(45 pgs tid 70 v 608) v1 from osd.1 10.252.24.25:6805/3572 -4144> 2013-04-03 10:47:34.644471 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 prepare_update pg_stats(45 pgs tid 70 v 608) v1 from osd.1 10.252.24.25:6805/3572 -4143> 2013-04-03 10:47:34.644476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 prepare_pg_stats pg_stats(45 pgs tid 70 v 608) v1 from osd.1 -4142> 2013-04-03 10:47:34.644482 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1108 MB used, 148 GB avail, 149 GB total, peers [0,3,4]/[])) -4141> 2013-04-03 10:47:34.869713 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.0 10.252.58.31:6802/2682 8 ==== pg_stats(51 pgs tid 413 v 608) v1 ==== 19135+0+0 (2091972375 0 0) 0x33e4000 con 0x3311580 -4140> 2013-04-03 10:47:34.869773 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 preprocess_query pg_stats(51 pgs tid 413 v 608) v1 from osd.0 10.252.58.31:6802/2682 -4139> 2013-04-03 10:47:34.869782 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 prepare_update pg_stats(51 pgs tid 413 v 608) v1 from osd.0 10.252.58.31:6802/2682 -4138> 2013-04-03 10:47:34.869786 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 prepare_pg_stats pg_stats(51 pgs tid 413 v 608) v1 from osd.0 -4137> 2013-04-03 10:47:34.869792 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 got osd.0 osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1141 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -4136> 2013-04-03 10:47:34.933200 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394130 encode_pending v394131 -4135> 2013-04-03 10:47:34.933234 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427569) queue_proposal bl 730 bytes; ctx = 0x2c3d760 -4134> 2013-04-03 10:47:34.933243 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427569) propose_queued 427570 730 bytes -4133> 2013-04-03 10:47:34.934671 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427569 fc 0 pn 17700 opn 0) v3 -- ?+0 0x300c500 -4132> 2013-04-03 10:47:34.934766 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 encode_pending v 29145 -4131> 2013-04-03 10:47:34.935036 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427522..427569) queue_proposal bl 36197 bytes; ctx = 0x1b91480 -4130> 2013-04-03 10:47:34.935045 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427522..427569) propose_new_value not active; proposal queued -4129> 2013-04-03 10:47:34.989598 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 46 ==== paxos(accept lc 427569 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (327939927 0 0) 0x300c500 con 0x208a840 -4128> 2013-04-03 10:47:34.989909 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427570 fc 0 pn 17700 opn 0) v3 -- ?+0 0x300c780 -4127> 2013-04-03 10:47:34.989928 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427570 fc 427522 pn 0 opn 0) v3 -- ?+0 0x300cc80 -4126> 2013-04-03 10:47:34.989973 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos -4125> 2013-04-03 10:47:34.990001 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos version 394131 summary v 394130 -4124> 2013-04-03 10:47:34.990017 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos latest full 393798 -4123> 2013-04-03 10:47:34.990035 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos applying incremental log 394131 2013-04-03 10:47:30.669033 mon.0 10.252.24.25:6789/0 220 : [INF] pgmap v29143: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 4466B/s wr, 0op/s -4122> 2013-04-03 10:47:34.990067 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos applying incremental log 394131 2013-04-03 10:47:33.934326 mon.0 10.252.24.25:6789/0 221 : [INF] pgmap v29144: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail -4121> 2013-04-03 10:47:34.990662 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 check_subs -4120> 2013-04-03 10:47:34.990704 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 create_pending v 394132 -4119> 2013-04-03 10:47:34.990728 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 _updated_log for mon.0 10.252.24.25:6789/0 -4118> 2013-04-03 10:47:34.990742 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 221) v1 -- ?+0 0x336b8c0 con 0x1bac160 -4117> 2013-04-03 10:47:34.990766 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427570) propose_queued 427571 36197 bytes -4116> 2013-04-03 10:47:34.992368 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427570 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2116280 -4115> 2013-04-03 10:47:34.992395 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 221) v1 ==== 0+0+0 (0 0 0) 0x336b8c0 con 0x1bac160 -4114> 2013-04-03 10:47:34.992403 7fcdffcdb700 10 handle_log_ack log(last 221) v1 -4113> 2013-04-03 10:47:34.992404 7fcdffcdb700 10 logged 2013-04-03 10:47:30.669033 mon.0 10.252.24.25:6789/0 220 : [INF] pgmap v29143: 240 pgs: 167 active+clean, 73 stale+active+clean; 256 MB data, 5661 MB used, 1067 GB / 1079 GB avail; 4466B/s wr, 0op/s -4112> 2013-04-03 10:47:34.992411 7fcdffcdb700 10 logged 2013-04-03 10:47:33.934326 mon.0 10.252.24.25:6789/0 221 : [INF] pgmap v29144: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail -4111> 2013-04-03 10:47:35.002899 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 47 ==== paxos(lease_ack lc 427570 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3157064837 0 0) 0x2116280 con 0x208a840 -4110> 2013-04-03 10:47:35.028311 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 48 ==== paxos(accept lc 427570 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (1442251650 0 0) 0x3250280 con 0x208a840 -4109> 2013-04-03 10:47:35.029411 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427571 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2116280 -4108> 2013-04-03 10:47:35.029429 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427571 fc 427522 pn 0 opn 0) v3 -- ?+0 0x300c500 -4107> 2013-04-03 10:47:35.029524 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29144 update_from_paxos applying incremental 29145 -4106> 2013-04-03 10:47:35.029757 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 v29145: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 926B/s wr, 0op/s -4105> 2013-04-03 10:47:35.029790 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 send_pg_creates to 0 pgs -4104> 2013-04-03 10:47:35.029795 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 update_logger -4103> 2013-04-03 10:47:35.029814 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 create_pending v 29146 -4102> 2013-04-03 10:47:35.029823 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 _updated_stats for osd.4 10.252.24.32:6800/13473 -4101> 2013-04-03 10:47:35.029830 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(0 pgs tid 64) v1 tid 8) v2 -- ?+0 0x24f8240 con 0x208a840 -4100> 2013-04-03 10:47:35.029839 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 _updated_stats for osd.1 10.252.24.25:6805/3572 -4099> 2013-04-03 10:47:35.029843 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(45 pgs tid 70) v1 -- ?+0 0x336b1c0 con 0x3311160 -4098> 2013-04-03 10:47:35.029864 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 _updated_stats for osd.0 10.252.58.31:6802/2682 -4097> 2013-04-03 10:47:35.029869 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6802/2682 -- pg_stats_ack(51 pgs tid 413) v1 -- ?+0 0x2e8b000 con 0x3311580 -4096> 2013-04-03 10:47:35.029887 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 check_osd_map already seen 608 >= 608 -4095> 2013-04-03 10:47:35.029891 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 update_logger -4094> 2013-04-03 10:47:35.029902 7fcdffcdb700 0 log [INF] : pgmap v29145: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 926B/s wr, 0op/s -4093> 2013-04-03 10:47:35.029903 7fcdffcdb700 10 send_log to self -4092> 2013-04-03 10:47:35.029904 7fcdffcdb700 10 log_queue is 1 last_log 222 sent 221 num 1 unsent 1 sending 1 -4091> 2013-04-03 10:47:35.029906 7fcdffcdb700 10 will send 2013-04-03 10:47:35.029903 mon.0 10.252.24.25:6789/0 222 : [INF] pgmap v29145: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 926B/s wr, 0op/s -4090> 2013-04-03 10:47:35.029912 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2117800 -4089> 2013-04-03 10:47:35.029961 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos -4088> 2013-04-03 10:47:35.029980 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos version 394131 summary v 394131 -4087> 2013-04-03 10:47:35.029999 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4086> 2013-04-03 10:47:35.030021 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2117800 con 0x1bac160 -4085> 2013-04-03 10:47:35.030043 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos -4084> 2013-04-03 10:47:35.030059 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos version 394131 summary v 394131 -4083> 2013-04-03 10:47:35.030068 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -4082> 2013-04-03 10:47:35.030079 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 preprocess_log log(1 entries) v1 from mon.0 -4081> 2013-04-03 10:47:35.030098 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -4080> 2013-04-03 10:47:35.030109 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 prepare_log log(1 entries) v1 from mon.0 -4079> 2013-04-03 10:47:35.030119 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 logging 2013-04-03 10:47:35.029903 mon.0 10.252.24.25:6789/0 222 : [INF] pgmap v29145: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 926B/s wr, 0op/s -4078> 2013-04-03 10:47:35.062006 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 49 ==== paxos(lease_ack lc 427571 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1051547646 0 0) 0x300c500 con 0x208a840 -4077> 2013-04-03 10:47:35.062050 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos -4076> 2013-04-03 10:47:35.062068 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 update_from_paxos version 394131 summary v 394131 -4075> 2013-04-03 10:47:35.062083 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4074> 2013-04-03 10:47:35.297097 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 50 ==== forward(pg_stats(71 pgs tid 52 v 608) v1) to leader v1 ==== 26867+0+0 (2171497347 0 0) 0x273a000 con 0x208a840 -4073> 2013-04-03 10:47:35.297112 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -4072> 2013-04-03 10:47:35.297121 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -4071> 2013-04-03 10:47:35.297154 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 preprocess_query pg_stats(71 pgs tid 52 v 608) v1 from osd.2 10.252.24.30:6801/5656 -4070> 2013-04-03 10:47:35.297162 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 prepare_update pg_stats(71 pgs tid 52 v 608) v1 from osd.2 10.252.24.30:6801/5656 -4069> 2013-04-03 10:47:35.297166 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 prepare_pg_stats pg_stats(71 pgs tid 52 v 608) v1 from osd.2 -4068> 2013-04-03 10:47:35.297172 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 got osd.2 osd_stat(1182 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1177 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -4067> 2013-04-03 10:47:35.600911 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 214 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140497 v188) v2 ==== 289+0+0 (2665980913 0 0) 0x20e3b00 con 0x2118420 -4066> 2013-04-03 10:47:35.600990 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140497 v188) v2 from mds.? 10.252.58.31:6800/10908 -4065> 2013-04-03 10:47:35.601004 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140497 v188) v2 -- ?+0 0x273a000 con 0x2118420 -4064> 2013-04-03 10:47:36.028507 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394131 encode_pending v394132 -4063> 2013-04-03 10:47:36.028536 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427571) queue_proposal bl 442 bytes; ctx = 0x2c3d860 -4062> 2013-04-03 10:47:36.028544 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427571) propose_queued 427572 442 bytes -4061> 2013-04-03 10:47:36.029804 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427571 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2116a00 -4060> 2013-04-03 10:47:36.029905 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 encode_pending v 29146 -4059> 2013-04-03 10:47:36.030116 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427522..427571) queue_proposal bl 26764 bytes; ctx = 0x2c3ddc0 -4058> 2013-04-03 10:47:36.030124 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427522..427571) propose_new_value not active; proposal queued -4057> 2013-04-03 10:47:36.052966 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 51 ==== paxos(accept lc 427571 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (4235757264 0 0) 0x2116a00 con 0x208a840 -4056> 2013-04-03 10:47:36.053289 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427572 fc 0 pn 17700 opn 0) v3 -- ?+0 0x300c500 -4055> 2013-04-03 10:47:36.053321 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427572 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250280 -4054> 2013-04-03 10:47:36.053357 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos -4053> 2013-04-03 10:47:36.053385 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos version 394132 summary v 394131 -4052> 2013-04-03 10:47:36.053400 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos latest full 393798 -4051> 2013-04-03 10:47:36.053419 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos applying incremental log 394132 2013-04-03 10:47:35.029903 mon.0 10.252.24.25:6789/0 222 : [INF] pgmap v29145: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 926B/s wr, 0op/s -4050> 2013-04-03 10:47:36.053464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 check_subs -4049> 2013-04-03 10:47:36.053500 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 create_pending v 394133 -4048> 2013-04-03 10:47:36.053523 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 _updated_log for mon.0 10.252.24.25:6789/0 -4047> 2013-04-03 10:47:36.053537 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 222) v1 -- ?+0 0x336be00 con 0x1bac160 -4046> 2013-04-03 10:47:36.053561 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427572) propose_queued 427573 26764 bytes -4045> 2013-04-03 10:47:36.055685 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427572 fc 0 pn 17700 opn 0) v3 -- ?+0 0x29b4000 -4044> 2013-04-03 10:47:36.055725 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 222) v1 ==== 0+0+0 (0 0 0) 0x336be00 con 0x1bac160 -4043> 2013-04-03 10:47:36.055733 7fcdffcdb700 10 handle_log_ack log(last 222) v1 -4042> 2013-04-03 10:47:36.055738 7fcdffcdb700 10 logged 2013-04-03 10:47:35.029903 mon.0 10.252.24.25:6789/0 222 : [INF] pgmap v29145: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 926B/s wr, 0op/s -4041> 2013-04-03 10:47:36.064940 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 52 ==== paxos(lease_ack lc 427572 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3591221898 0 0) 0x29b4000 con 0x208a840 -4040> 2013-04-03 10:47:36.090320 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 53 ==== paxos(accept lc 427572 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (4146192191 0 0) 0x3250280 con 0x208a840 -4039> 2013-04-03 10:47:36.091386 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427573 fc 0 pn 17700 opn 0) v3 -- ?+0 0x29b4000 -4038> 2013-04-03 10:47:36.091410 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427573 fc 427522 pn 0 opn 0) v3 -- ?+0 0x2116a00 -4037> 2013-04-03 10:47:36.091497 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29145 update_from_paxos applying incremental 29146 -4036> 2013-04-03 10:47:36.091681 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 v29146: 240 pgs: 240 active+clean; 256 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 1922B/s wr, 0op/s -4035> 2013-04-03 10:47:36.091710 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 send_pg_creates to 0 pgs -4034> 2013-04-03 10:47:36.091714 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 update_logger -4033> 2013-04-03 10:47:36.091729 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 create_pending v 29147 -4032> 2013-04-03 10:47:36.091739 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 _updated_stats for osd.2 10.252.24.30:6801/5656 -4031> 2013-04-03 10:47:36.091745 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(71 pgs tid 52) v1 tid 9) v2 -- ?+0 0x33e4000 con 0x208a840 -4030> 2013-04-03 10:47:36.091776 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 check_osd_map already seen 608 >= 608 -4029> 2013-04-03 10:47:36.091780 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 update_logger -4028> 2013-04-03 10:47:36.091791 7fcdffcdb700 0 log [INF] : pgmap v29146: 240 pgs: 240 active+clean; 256 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 1922B/s wr, 0op/s -4027> 2013-04-03 10:47:36.091792 7fcdffcdb700 10 send_log to self -4026> 2013-04-03 10:47:36.091793 7fcdffcdb700 10 log_queue is 1 last_log 223 sent 222 num 1 unsent 1 sending 1 -4025> 2013-04-03 10:47:36.091795 7fcdffcdb700 10 will send 2013-04-03 10:47:36.091792 mon.0 10.252.24.25:6789/0 223 : [INF] pgmap v29146: 240 pgs: 240 active+clean; 256 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 1922B/s wr, 0op/s -4024> 2013-04-03 10:47:36.091801 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2226c00 -4023> 2013-04-03 10:47:36.091851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos -4022> 2013-04-03 10:47:36.091869 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos version 394132 summary v 394132 -4021> 2013-04-03 10:47:36.091889 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4020> 2013-04-03 10:47:36.091910 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2226c00 con 0x1bac160 -4019> 2013-04-03 10:47:36.091932 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos -4018> 2013-04-03 10:47:36.091947 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos version 394132 summary v 394132 -4017> 2013-04-03 10:47:36.091957 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -4016> 2013-04-03 10:47:36.091969 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 preprocess_log log(1 entries) v1 from mon.0 -4015> 2013-04-03 10:47:36.091989 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -4014> 2013-04-03 10:47:36.092003 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 prepare_log log(1 entries) v1 from mon.0 -4013> 2013-04-03 10:47:36.092013 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 logging 2013-04-03 10:47:36.091792 mon.0 10.252.24.25:6789/0 223 : [INF] pgmap v29146: 240 pgs: 240 active+clean; 256 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 1922B/s wr, 0op/s -4012> 2013-04-03 10:47:36.124011 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 54 ==== paxos(lease_ack lc 427573 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3192646399 0 0) 0x2116a00 con 0x208a840 -4011> 2013-04-03 10:47:36.124056 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos -4010> 2013-04-03 10:47:36.124073 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 update_from_paxos version 394132 summary v 394132 -4009> 2013-04-03 10:47:36.124088 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -4008> 2013-04-03 10:47:36.947529 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 55 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 212 v188) v2) to leader v1 ==== 560+0+0 (275625247 0 0) 0x2145580 con 0x208a840 -4007> 2013-04-03 10:47:36.947551 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -4006> 2013-04-03 10:47:36.947561 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ce2340 from 10.252.24.88:6789/0 -4005> 2013-04-03 10:47:36.947618 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 212 v188) v2 from mds.0 10.252.24.25:6802/3144 -4004> 2013-04-03 10:47:36.947633 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 212 v188) v2 tid 10) v2 -- ?+0 0x24f8240 con 0x208a840 -4003> 2013-04-03 10:47:37.090505 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394132 encode_pending v394133 -4002> 2013-04-03 10:47:37.090537 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427573) queue_proposal bl 443 bytes; ctx = 0x2c3df50 -4001> 2013-04-03 10:47:37.090545 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427573) propose_queued 427574 443 bytes -4000> 2013-04-03 10:47:37.091811 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427573 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2116000 -3999> 2013-04-03 10:47:37.115074 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 56 ==== paxos(accept lc 427573 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (3576352708 0 0) 0x2116000 con 0x208a840 -3998> 2013-04-03 10:47:37.115409 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427574 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2116a00 -3997> 2013-04-03 10:47:37.115438 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427574 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250280 -3996> 2013-04-03 10:47:37.115476 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos -3995> 2013-04-03 10:47:37.115504 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394132 -3994> 2013-04-03 10:47:37.115518 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos latest full 393798 -3993> 2013-04-03 10:47:37.115538 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos applying incremental log 394133 2013-04-03 10:47:36.091792 mon.0 10.252.24.25:6789/0 223 : [INF] pgmap v29146: 240 pgs: 240 active+clean; 256 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 1922B/s wr, 0op/s -3992> 2013-04-03 10:47:37.115585 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 check_subs -3991> 2013-04-03 10:47:37.115621 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 create_pending v 394134 -3990> 2013-04-03 10:47:37.115644 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 _updated_log for mon.0 10.252.24.25:6789/0 -3989> 2013-04-03 10:47:37.115657 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 223) v1 -- ?+0 0x336be00 con 0x1bac160 -3988> 2013-04-03 10:47:37.115707 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos -3987> 2013-04-03 10:47:37.115724 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 -3986> 2013-04-03 10:47:37.115743 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3985> 2013-04-03 10:47:37.115764 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 223) v1 ==== 0+0+0 (0 0 0) 0x336be00 con 0x1bac160 -3984> 2013-04-03 10:47:37.115772 7fcdffcdb700 10 handle_log_ack log(last 223) v1 -3983> 2013-04-03 10:47:37.115775 7fcdffcdb700 10 logged 2013-04-03 10:47:36.091792 mon.0 10.252.24.25:6789/0 223 : [INF] pgmap v29146: 240 pgs: 240 active+clean; 256 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 1922B/s wr, 0op/s -3982> 2013-04-03 10:47:37.126992 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 57 ==== paxos(lease_ack lc 427574 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (2249629551 0 0) 0x3250280 con 0x208a840 -3981> 2013-04-03 10:47:37.127030 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos -3980> 2013-04-03 10:47:37.127047 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 -3979> 2013-04-03 10:47:37.127062 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3978> 2013-04-03 10:47:37.436564 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 10 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 3 v188) v2 ==== 289+0+0 (704987183 0 0) 0x2a83dc0 con 0x3311000 -3977> 2013-04-03 10:47:37.436641 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 3 v188) v2 from mds.? 10.252.24.88:6800/18040 -3976> 2013-04-03 10:47:37.436655 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 3 v188) v2 -- ?+0 0x2145580 con 0x3311000 -3975> 2013-04-03 10:47:38.385720 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 check_down_pgs -3974> 2013-04-03 10:47:38.385749 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 v29146: 240 pgs: 240 active+clean; 256 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 1922B/s wr, 0op/s -3973> 2013-04-03 10:47:38.385779 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -3972> 2013-04-03 10:47:38.385804 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in -3971> 2013-04-03 10:47:38.385842 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 -3970> 2013-04-03 10:47:38.385874 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos -3969> 2013-04-03 10:47:38.385893 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 -3968> 2013-04-03 10:47:38.385903 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 log -3967> 2013-04-03 10:47:38.385920 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3966> 2013-04-03 10:47:38.385939 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth -3965> 2013-04-03 10:47:39.601026 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 215 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140498 v188) v2 ==== 289+0+0 (4216611446 0 0) 0x2f7a8c0 con 0x2118420 -3964> 2013-04-03 10:47:39.601113 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140498 v188) v2 from mds.? 10.252.58.31:6800/10908 -3963> 2013-04-03 10:47:39.601132 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140498 v188) v2 -- ?+0 0x2a83dc0 con 0x2118420 -3962> 2013-04-03 10:47:40.115518 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427574 fc 427522 pn 0 opn 0) v3 -- ?+0 0x2116c80 -3961> 2013-04-03 10:47:40.116368 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 58 ==== paxos(lease_ack lc 427574 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3142840274 0 0) 0x2116c80 con 0x208a840 -3960> 2013-04-03 10:47:40.116444 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos -3959> 2013-04-03 10:47:40.116465 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 -3958> 2013-04-03 10:47:40.116483 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3957> 2013-04-03 10:47:40.947606 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 59 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 213 v188) v2) to leader v1 ==== 560+0+0 (421817116 0 0) 0x20e3b00 con 0x208a840 -3956> 2013-04-03 10:47:40.947631 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -3955> 2013-04-03 10:47:40.947642 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2ce1580 from 10.252.24.88:6789/0 -3954> 2013-04-03 10:47:40.947699 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 213 v188) v2 from mds.0 10.252.24.25:6802/3144 -3953> 2013-04-03 10:47:40.947711 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 213 v188) v2 tid 11) v2 -- ?+0 0x33e4000 con 0x208a840 -3952> 2013-04-03 10:47:41.436620 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 11 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 4 v188) v2 ==== 289+0+0 (3255598172 0 0) 0x2145dc0 con 0x3311000 -3951> 2013-04-03 10:47:41.436704 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 4 v188) v2 from mds.? 10.252.24.88:6800/18040 -3950> 2013-04-03 10:47:41.436718 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 4 v188) v2 -- ?+0 0x20e3b00 con 0x3311000 -3949> 2013-04-03 10:47:43.115586 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427574 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250780 -3948> 2013-04-03 10:47:43.116393 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 60 ==== paxos(lease_ack lc 427574 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3659118585 0 0) 0x3250780 con 0x208a840 -3947> 2013-04-03 10:47:43.116471 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos -3946> 2013-04-03 10:47:43.116492 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 -3945> 2013-04-03 10:47:43.116511 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3944> 2013-04-03 10:47:43.386096 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 v29146: 240 pgs: 240 active+clean; 256 MB data, 5670 MB used, 1067 GB / 1079 GB avail; 1922B/s wr, 0op/s -3943> 2013-04-03 10:47:43.386139 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -3942> 2013-04-03 10:47:43.386163 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in -3941> 2013-04-03 10:47:43.386206 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 -3940> 2013-04-03 10:47:43.386238 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos -3939> 2013-04-03 10:47:43.386257 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 -3938> 2013-04-03 10:47:43.386267 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 log -3937> 2013-04-03 10:47:43.386286 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3936> 2013-04-03 10:47:43.386306 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth -3935> 2013-04-03 10:47:43.601140 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 216 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140499 v188) v2 ==== 289+0+0 (3627736331 0 0) 0x2f792c0 con 0x2118420 -3934> 2013-04-03 10:47:43.601224 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140499 v188) v2 from mds.? 10.252.58.31:6800/10908 -3933> 2013-04-03 10:47:43.601239 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140499 v188) v2 -- ?+0 0x2145dc0 con 0x2118420 -3932> 2013-04-03 10:47:44.947798 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 61 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 214 v188) v2) to leader v1 ==== 560+0+0 (927161120 0 0) 0x2f7a8c0 con 0x208a840 -3931> 2013-04-03 10:47:44.947820 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -3930> 2013-04-03 10:47:44.947831 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a848c0 from 10.252.24.88:6789/0 -3929> 2013-04-03 10:47:44.947887 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 214 v188) v2 from mds.0 10.252.24.25:6802/3144 -3928> 2013-04-03 10:47:44.947900 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 214 v188) v2 tid 12) v2 -- ?+0 0x24f8240 con 0x208a840 -3927> 2013-04-03 10:47:45.293288 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 62 ==== forward(pg_stats(1 pgs tid 53 v 608) v1) to leader v1 ==== 757+0+0 (886919364 0 0) 0x2f792c0 con 0x208a840 -3926> 2013-04-03 10:47:45.293316 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -3925> 2013-04-03 10:47:45.293327 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x24f8240 from 10.252.24.88:6789/0 -3924> 2013-04-03 10:47:45.293382 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 preprocess_query pg_stats(1 pgs tid 53 v 608) v1 from osd.2 10.252.24.30:6801/5656 -3923> 2013-04-03 10:47:45.293390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 prepare_update pg_stats(1 pgs tid 53 v 608) v1 from osd.2 10.252.24.30:6801/5656 -3922> 2013-04-03 10:47:45.293397 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 prepare_pg_stats pg_stats(1 pgs tid 53 v 608) v1 from osd.2 -3921> 2013-04-03 10:47:45.293404 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 got osd.2 osd_stat(1181 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1182 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -3920> 2013-04-03 10:47:45.343587 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 encode_pending v 29147 -3919> 2013-04-03 10:47:45.343632 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427574) queue_proposal bl 654 bytes; ctx = 0x2c3ddf0 -3918> 2013-04-03 10:47:45.343640 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427574) propose_queued 427575 654 bytes -3917> 2013-04-03 10:47:45.344478 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427574 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3250c80 -3916> 2013-04-03 10:47:45.436741 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 12 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 5 v188) v2 ==== 289+0+0 (3781346081 0 0) 0x2145580 con 0x3311000 -3915> 2013-04-03 10:47:45.437121 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 5 v188) v2 from mds.? 10.252.24.88:6800/18040 -3914> 2013-04-03 10:47:45.437140 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 5 v188) v2 -- ?+0 0x2f792c0 con 0x3311000 -3913> 2013-04-03 10:47:45.437507 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 63 ==== paxos(accept lc 427574 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (3587432912 0 0) 0x3250c80 con 0x208a840 -3912> 2013-04-03 10:47:45.437826 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427575 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3250780 -3911> 2013-04-03 10:47:45.437845 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427575 fc 427522 pn 0 opn 0) v3 -- ?+0 0x2116c80 -3910> 2013-04-03 10:47:45.437927 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29146 update_from_paxos applying incremental 29147 -3909> 2013-04-03 10:47:45.437950 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 v29147: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 7870B/s wr, 0op/s -3908> 2013-04-03 10:47:45.437971 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 send_pg_creates to 0 pgs -3907> 2013-04-03 10:47:45.438251 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 update_logger -3906> 2013-04-03 10:47:45.438269 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 create_pending v 29148 -3905> 2013-04-03 10:47:45.438281 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 _updated_stats for osd.2 10.252.24.30:6801/5656 -3904> 2013-04-03 10:47:45.438289 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 53) v1 tid 13) v2 -- ?+0 0x33e4000 con 0x208a840 -3903> 2013-04-03 10:47:45.438300 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 check_osd_map already seen 608 >= 608 -3902> 2013-04-03 10:47:45.438304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 update_logger -3901> 2013-04-03 10:47:45.438331 7fcdffcdb700 0 log [INF] : pgmap v29147: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 7870B/s wr, 0op/s -3900> 2013-04-03 10:47:45.438334 7fcdffcdb700 10 send_log to self -3899> 2013-04-03 10:47:45.438340 7fcdffcdb700 10 log_queue is 1 last_log 224 sent 223 num 1 unsent 1 sending 1 -3898> 2013-04-03 10:47:45.438342 7fcdffcdb700 10 will send 2013-04-03 10:47:45.438334 mon.0 10.252.24.25:6789/0 224 : [INF] pgmap v29147: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 7870B/s wr, 0op/s -3897> 2013-04-03 10:47:45.438350 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2226c00 -3896> 2013-04-03 10:47:45.438404 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos -3895> 2013-04-03 10:47:45.438424 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 -3894> 2013-04-03 10:47:45.438443 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3893> 2013-04-03 10:47:45.438470 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2226c00 con 0x1bac160 -3892> 2013-04-03 10:47:45.438492 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos -3891> 2013-04-03 10:47:45.438508 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 -3890> 2013-04-03 10:47:45.438517 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3889> 2013-04-03 10:47:45.438531 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 preprocess_log log(1 entries) v1 from mon.0 -3888> 2013-04-03 10:47:45.438555 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3887> 2013-04-03 10:47:45.438568 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 prepare_log log(1 entries) v1 from mon.0 -3886> 2013-04-03 10:47:45.438578 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 logging 2013-04-03 10:47:45.438334 mon.0 10.252.24.25:6789/0 224 : [INF] pgmap v29147: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 7870B/s wr, 0op/s -3885> 2013-04-03 10:47:45.461578 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 64 ==== paxos(lease_ack lc 427575 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (4119203230 0 0) 0x2116c80 con 0x208a840 -3884> 2013-04-03 10:47:45.461668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos -3883> 2013-04-03 10:47:45.461690 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 update_from_paxos version 394133 summary v 394133 -3882> 2013-04-03 10:47:45.461707 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3881> 2013-04-03 10:47:46.437683 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394133 encode_pending v394134 -3880> 2013-04-03 10:47:46.437714 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427575) queue_proposal bl 443 bytes; ctx = 0x2c3dde0 -3879> 2013-04-03 10:47:46.437722 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427575) propose_queued 427576 443 bytes -3878> 2013-04-03 10:47:46.438814 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427575 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3434c80 -3877> 2013-04-03 10:47:46.460885 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 65 ==== paxos(accept lc 427575 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (2029930381 0 0) 0x3434c80 con 0x208a840 -3876> 2013-04-03 10:47:46.461179 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427576 fc 0 pn 17700 opn 0) v3 -- ?+0 0x2116c80 -3875> 2013-04-03 10:47:46.461197 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427576 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3250c80 -3874> 2013-04-03 10:47:46.461233 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos -3873> 2013-04-03 10:47:46.461262 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394133 -3872> 2013-04-03 10:47:46.461277 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos latest full 393798 -3871> 2013-04-03 10:47:46.461294 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos applying incremental log 394134 2013-04-03 10:47:45.438334 mon.0 10.252.24.25:6789/0 224 : [INF] pgmap v29147: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 7870B/s wr, 0op/s -3870> 2013-04-03 10:47:46.461345 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 check_subs -3869> 2013-04-03 10:47:46.461381 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 create_pending v 394135 -3868> 2013-04-03 10:47:46.461404 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 _updated_log for mon.0 10.252.24.25:6789/0 -3867> 2013-04-03 10:47:46.461417 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 224) v1 -- ?+0 0x2224540 con 0x1bac160 -3866> 2013-04-03 10:47:46.461470 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos -3865> 2013-04-03 10:47:46.461486 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 -3864> 2013-04-03 10:47:46.461505 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3863> 2013-04-03 10:47:46.461527 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 224) v1 ==== 0+0+0 (0 0 0) 0x2224540 con 0x1bac160 -3862> 2013-04-03 10:47:46.461535 7fcdffcdb700 10 handle_log_ack log(last 224) v1 -3861> 2013-04-03 10:47:46.461538 7fcdffcdb700 10 logged 2013-04-03 10:47:45.438334 mon.0 10.252.24.25:6789/0 224 : [INF] pgmap v29147: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 7870B/s wr, 0op/s -3860> 2013-04-03 10:47:46.474129 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 66 ==== paxos(lease_ack lc 427576 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1729586873 0 0) 0x3250c80 con 0x208a840 -3859> 2013-04-03 10:47:46.474165 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos -3858> 2013-04-03 10:47:46.474182 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 -3857> 2013-04-03 10:47:46.474200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3856> 2013-04-03 10:47:47.601274 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 217 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140500 v188) v2 ==== 289+0+0 (808706424 0 0) 0x2f79dc0 con 0x2118420 -3855> 2013-04-03 10:47:47.601359 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140500 v188) v2 from mds.? 10.252.58.31:6800/10908 -3854> 2013-04-03 10:47:47.601379 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140500 v188) v2 -- ?+0 0x2145580 con 0x2118420 -3853> 2013-04-03 10:47:48.386426 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 check_down_pgs -3852> 2013-04-03 10:47:48.386457 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 v29147: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 7870B/s wr, 0op/s -3851> 2013-04-03 10:47:48.386486 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -3850> 2013-04-03 10:47:48.386510 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in -3849> 2013-04-03 10:47:48.386549 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 -3848> 2013-04-03 10:47:48.386580 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos -3847> 2013-04-03 10:47:48.386600 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 -3846> 2013-04-03 10:47:48.386610 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 log -3845> 2013-04-03 10:47:48.386628 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3844> 2013-04-03 10:47:48.386652 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth -3843> 2013-04-03 10:47:48.948036 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 67 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 215 v188) v2) to leader v1 ==== 560+0+0 (3929431669 0 0) 0x2a84600 con 0x208a840 -3842> 2013-04-03 10:47:48.948058 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -3841> 2013-04-03 10:47:48.948069 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2a528c0 from 10.252.24.88:6789/0 -3840> 2013-04-03 10:47:48.948125 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 215 v188) v2 from mds.0 10.252.24.25:6802/3144 -3839> 2013-04-03 10:47:48.948137 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 215 v188) v2 tid 14) v2 -- ?+0 0x24f8240 con 0x208a840 -3838> 2013-04-03 10:47:49.436879 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 13 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 6 v188) v2 ==== 289+0+0 (2228290214 0 0) 0x2a84340 con 0x3311000 -3837> 2013-04-03 10:47:49.436964 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 6 v188) v2 from mds.? 10.252.24.88:6800/18040 -3836> 2013-04-03 10:47:49.436981 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 6 v188) v2 -- ?+0 0x2a84600 con 0x3311000 -3835> 2013-04-03 10:47:49.461276 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427576 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3434a00 -3834> 2013-04-03 10:47:49.462042 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 68 ==== paxos(lease_ack lc 427576 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1767907080 0 0) 0x3434a00 con 0x208a840 -3833> 2013-04-03 10:47:49.462114 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos -3832> 2013-04-03 10:47:49.462135 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 -3831> 2013-04-03 10:47:49.462153 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3830> 2013-04-03 10:47:51.601374 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 218 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140501 v188) v2 ==== 289+0+0 (324886021 0 0) 0x2f79b00 con 0x2118420 -3829> 2013-04-03 10:47:51.601454 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140501 v188) v2 from mds.? 10.252.58.31:6800/10908 -3828> 2013-04-03 10:47:51.601469 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140501 v188) v2 -- ?+0 0x2a84340 con 0x2118420 -3827> 2013-04-03 10:47:52.461333 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427576 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3434780 -3826> 2013-04-03 10:47:52.461984 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 69 ==== paxos(lease_ack lc 427576 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (62815001 0 0) 0x3434780 con 0x208a840 -3825> 2013-04-03 10:47:52.462062 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos -3824> 2013-04-03 10:47:52.462083 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 -3823> 2013-04-03 10:47:52.462102 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3822> 2013-04-03 10:47:52.948047 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 70 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 216 v188) v2) to leader v1 ==== 560+0+0 (1483395147 0 0) 0x2f79dc0 con 0x208a840 -3821> 2013-04-03 10:47:52.948069 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -3820> 2013-04-03 10:47:52.948080 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2f79580 from 10.252.24.88:6789/0 -3819> 2013-04-03 10:47:52.948137 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 216 v188) v2 from mds.0 10.252.24.25:6802/3144 -3818> 2013-04-03 10:47:52.948150 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 216 v188) v2 tid 15) v2 -- ?+0 0x33e4000 con 0x208a840 -3817> 2013-04-03 10:47:53.386779 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 v29147: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 7870B/s wr, 0op/s -3816> 2013-04-03 10:47:53.386822 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -3815> 2013-04-03 10:47:53.386846 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in -3814> 2013-04-03 10:47:53.386892 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 -3813> 2013-04-03 10:47:53.386924 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos -3812> 2013-04-03 10:47:53.386944 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 -3811> 2013-04-03 10:47:53.386953 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 log -3810> 2013-04-03 10:47:53.386969 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3809> 2013-04-03 10:47:53.386989 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth -3808> 2013-04-03 10:47:53.436946 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 14 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 7 v188) v2 ==== 289+0+0 (2814361051 0 0) 0x273adc0 con 0x3311000 -3807> 2013-04-03 10:47:53.437002 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 7 v188) v2 from mds.? 10.252.24.88:6800/18040 -3806> 2013-04-03 10:47:53.437015 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 7 v188) v2 -- ?+0 0x2f79dc0 con 0x3311000 -3805> 2013-04-03 10:47:54.646462 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 9 ==== pg_stats(1 pgs tid 71 v 608) v1 ==== 477+0+0 (1921778279 0 0) 0x33e46c0 con 0x3311160 -3804> 2013-04-03 10:47:54.646533 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 preprocess_query pg_stats(1 pgs tid 71 v 608) v1 from osd.1 10.252.24.25:6805/3572 -3803> 2013-04-03 10:47:54.646543 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 prepare_update pg_stats(1 pgs tid 71 v 608) v1 from osd.1 10.252.24.25:6805/3572 -3802> 2013-04-03 10:47:54.646547 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 prepare_pg_stats pg_stats(1 pgs tid 71 v 608) v1 from osd.1 -3801> 2013-04-03 10:47:54.646554 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) -3800> 2013-04-03 10:47:54.696755 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 encode_pending v 29148 -3799> 2013-04-03 10:47:54.696791 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427576) queue_proposal bl 646 bytes; ctx = 0x330b020 -3798> 2013-04-03 10:47:54.696799 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427576) propose_queued 427577 646 bytes -3797> 2013-04-03 10:47:54.697775 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427576 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3434500 -3796> 2013-04-03 10:47:54.726083 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 71 ==== paxos(accept lc 427576 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (2040504135 0 0) 0x3434500 con 0x208a840 -3795> 2013-04-03 10:47:54.726420 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427577 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3434780 -3794> 2013-04-03 10:47:54.726455 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427577 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3434a00 -3793> 2013-04-03 10:47:54.726528 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29147 update_from_paxos applying incremental 29148 -3792> 2013-04-03 10:47:54.726548 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 v29148: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4607B/s wr, 0op/s -3791> 2013-04-03 10:47:54.726567 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 send_pg_creates to 0 pgs -3790> 2013-04-03 10:47:54.726569 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 update_logger -3789> 2013-04-03 10:47:54.726579 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 create_pending v 29149 -3788> 2013-04-03 10:47:54.726589 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 _updated_stats for osd.1 10.252.24.25:6805/3572 -3787> 2013-04-03 10:47:54.726594 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(1 pgs tid 71) v1 -- ?+0 0x2224540 con 0x3311160 -3786> 2013-04-03 10:47:54.726605 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 check_osd_map already seen 608 >= 608 -3785> 2013-04-03 10:47:54.726608 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 update_logger -3784> 2013-04-03 10:47:54.726618 7fcdffcdb700 0 log [INF] : pgmap v29148: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4607B/s wr, 0op/s -3783> 2013-04-03 10:47:54.726620 7fcdffcdb700 10 send_log to self -3782> 2013-04-03 10:47:54.726621 7fcdffcdb700 10 log_queue is 1 last_log 225 sent 224 num 1 unsent 1 sending 1 -3781> 2013-04-03 10:47:54.726622 7fcdffcdb700 10 will send 2013-04-03 10:47:54.726619 mon.0 10.252.24.25:6789/0 225 : [INF] pgmap v29148: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4607B/s wr, 0op/s -3780> 2013-04-03 10:47:54.726629 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2874800 -3779> 2013-04-03 10:47:54.726680 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos -3778> 2013-04-03 10:47:54.726701 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 -3777> 2013-04-03 10:47:54.726719 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3776> 2013-04-03 10:47:54.726742 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2874800 con 0x1bac160 -3775> 2013-04-03 10:47:54.726766 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos -3774> 2013-04-03 10:47:54.726781 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 -3773> 2013-04-03 10:47:54.726789 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3772> 2013-04-03 10:47:54.726801 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 preprocess_log log(1 entries) v1 from mon.0 -3771> 2013-04-03 10:47:54.726825 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3770> 2013-04-03 10:47:54.726837 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 prepare_log log(1 entries) v1 from mon.0 -3769> 2013-04-03 10:47:54.726847 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 logging 2013-04-03 10:47:54.726619 mon.0 10.252.24.25:6789/0 225 : [INF] pgmap v29148: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4607B/s wr, 0op/s -3768> 2013-04-03 10:47:54.738075 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 72 ==== paxos(lease_ack lc 427577 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1151097011 0 0) 0x3434a00 con 0x208a840 -3767> 2013-04-03 10:47:54.738170 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos -3766> 2013-04-03 10:47:54.738191 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 update_from_paxos version 394134 summary v 394134 -3765> 2013-04-03 10:47:54.738209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3764> 2013-04-03 10:47:55.601506 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 219 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140502 v188) v2 ==== 289+0+0 (1995349890 0 0) 0x2cc5600 con 0x2118420 -3763> 2013-04-03 10:47:55.601593 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140502 v188) v2 from mds.? 10.252.58.31:6800/10908 -3762> 2013-04-03 10:47:55.601609 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140502 v188) v2 -- ?+0 0x273adc0 con 0x2118420 -3761> 2013-04-03 10:47:55.726262 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394134 encode_pending v394135 -3760> 2013-04-03 10:47:55.726296 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427577) queue_proposal bl 443 bytes; ctx = 0x2c3def0 -3759> 2013-04-03 10:47:55.726304 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427577) propose_queued 427578 443 bytes -3758> 2013-04-03 10:47:55.727291 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427577 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3434280 -3757> 2013-04-03 10:47:55.762525 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 73 ==== paxos(accept lc 427577 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (539351697 0 0) 0x3434280 con 0x208a840 -3756> 2013-04-03 10:47:55.762903 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427578 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3434a00 -3755> 2013-04-03 10:47:55.762926 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427578 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3434500 -3754> 2013-04-03 10:47:55.762971 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos -3753> 2013-04-03 10:47:55.763009 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos version 394135 summary v 394134 -3752> 2013-04-03 10:47:55.763027 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos latest full 393798 -3751> 2013-04-03 10:47:55.763046 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos applying incremental log 394135 2013-04-03 10:47:54.726619 mon.0 10.252.24.25:6789/0 225 : [INF] pgmap v29148: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4607B/s wr, 0op/s -3750> 2013-04-03 10:47:55.763098 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 check_subs -3749> 2013-04-03 10:47:55.763135 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 create_pending v 394136 -3748> 2013-04-03 10:47:55.763158 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 _updated_log for mon.0 10.252.24.25:6789/0 -3747> 2013-04-03 10:47:55.763171 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 225) v1 -- ?+0 0x2224700 con 0x1bac160 -3746> 2013-04-03 10:47:55.763224 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos -3745> 2013-04-03 10:47:55.763241 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos version 394135 summary v 394135 -3744> 2013-04-03 10:47:55.763261 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3743> 2013-04-03 10:47:55.763283 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 225) v1 ==== 0+0+0 (0 0 0) 0x2224700 con 0x1bac160 -3742> 2013-04-03 10:47:55.763291 7fcdffcdb700 10 handle_log_ack log(last 225) v1 -3741> 2013-04-03 10:47:55.763293 7fcdffcdb700 10 logged 2013-04-03 10:47:54.726619 mon.0 10.252.24.25:6789/0 225 : [INF] pgmap v29148: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4607B/s wr, 0op/s -3740> 2013-04-03 10:47:55.775733 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 74 ==== paxos(lease_ack lc 427578 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (2839610900 0 0) 0x3434500 con 0x208a840 -3739> 2013-04-03 10:47:55.775769 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos -3738> 2013-04-03 10:47:55.775786 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos version 394135 summary v 394135 -3737> 2013-04-03 10:47:55.775801 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3736> 2013-04-03 10:47:56.948230 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 75 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 217 v188) v2) to leader v1 ==== 560+0+0 (1094263427 0 0) 0x2f79b00 con 0x208a840 -3735> 2013-04-03 10:47:56.948254 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -3734> 2013-04-03 10:47:56.948265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x2cc4000 from 10.252.24.88:6789/0 -3733> 2013-04-03 10:47:56.948323 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 217 v188) v2 from mds.0 10.252.24.25:6802/3144 -3732> 2013-04-03 10:47:56.948335 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 217 v188) v2 tid 16) v2 -- ?+0 0x33e46c0 con 0x208a840 -3731> 2013-04-03 10:47:57.437106 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 15 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 8 v188) v2 ==== 289+0+0 (1362037937 0 0) 0x273bb80 con 0x3311000 -3730> 2013-04-03 10:47:57.437186 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 8 v188) v2 from mds.? 10.252.24.88:6800/18040 -3729> 2013-04-03 10:47:57.437200 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 8 v188) v2 -- ?+0 0x2f79b00 con 0x3311000 -3728> 2013-04-03 10:47:58.387109 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 check_down_pgs -3727> 2013-04-03 10:47:58.387141 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29148 v29148: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4607B/s wr, 0op/s -3726> 2013-04-03 10:47:58.387172 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -3725> 2013-04-03 10:47:58.387196 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in -3724> 2013-04-03 10:47:58.387235 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 -3723> 2013-04-03 10:47:58.387266 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos -3722> 2013-04-03 10:47:58.387286 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos version 394135 summary v 394135 -3721> 2013-04-03 10:47:58.387296 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 log -3720> 2013-04-03 10:47:58.387327 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3719> 2013-04-03 10:47:58.387347 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth -3718> 2013-04-03 10:47:58.763004 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427578 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3434000 -3717> 2013-04-03 10:47:58.763764 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 76 ==== paxos(lease_ack lc 427578 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (2143876619 0 0) 0x3434000 con 0x208a840 -3716> 2013-04-03 10:47:58.763840 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos -3715> 2013-04-03 10:47:58.763862 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394135 update_from_paxos version 394135 summary v 394135 -3714> 2013-04-03 10:47:58.763879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3713> 2013-04-03 10:47:59.107977 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) -3712> 2013-04-03 10:47:59.108026 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader -3711> 2013-04-03 10:47:59.601602 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 220 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140503 v188) v2 ==== 289+0+0 (1434460415 0 0) 0x2cc4580 con 0x2118420 -3710> 2013-04-03 10:47:59.601685 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140503 v188) v2 from mds.? 10.252.58.31:6800/10908 -3709> 2013-04-03 10:47:59.601699 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140503 v188) v2 -- ?+0 0x273bb80 con 0x2118420 -3708> 2013-04-03 10:48:00.948235 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 77 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 218 v188) v2) to leader v1 ==== 560+0+0 (3268137698 0 0) 0x2cc5600 con 0x208a840 -3707> 2013-04-03 10:48:00.948258 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -3706> 2013-04-03 10:48:00.948268 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc42c0 from 10.252.24.88:6789/0 -3705> 2013-04-03 10:48:00.948326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 218 v188) v2 from mds.0 10.252.24.25:6802/3144 -3704> 2013-04-03 10:48:00.948337 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 218 v188) v2 tid 17) v2 -- ?+0 0x33e4000 con 0x208a840 -3703> 2013-04-03 10:48:01.437228 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 16 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 9 v188) v2 ==== 289+0+0 (1916908492 0 0) 0x2a52600 con 0x3311000 -3702> 2013-04-03 10:48:01.437312 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 9 v188) v2 from mds.? 10.252.24.88:6800/18040 -3701> 2013-04-03 10:48:01.437335 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 9 v188) v2 -- ?+0 0x2cc5600 con 0x3311000 -3700> 2013-04-03 10:48:01.763086 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427578 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3450c80 -3699> 2013-04-03 10:48:01.763901 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 78 ==== paxos(lease_ack lc 427578 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3761305779 0 0) 0x3450c80 con 0x208a840 -3698> 2013-04-03 10:48:01.763976 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos -3697> 2013-04-03 10:48:01.763996 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos version 394135 summary v 394135 -3696> 2013-04-03 10:48:01.764014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3695> 2013-04-03 10:48:03.387472 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29148 v29148: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 4607B/s wr, 0op/s -3694> 2013-04-03 10:48:03.387516 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -3693> 2013-04-03 10:48:03.387539 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -3692> 2013-04-03 10:48:03.387584 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -3691> 2013-04-03 10:48:03.387626 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos -3690> 2013-04-03 10:48:03.387644 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos version 394135 summary v 394135 -3689> 2013-04-03 10:48:03.387652 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394135 log -3688> 2013-04-03 10:48:03.387669 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3687> 2013-04-03 10:48:03.387688 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -3686> 2013-04-03 10:48:03.602251 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 221 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140504 v188) v2 ==== 289+0+0 (2735772053 0 0) 0x2cc5340 con 0x2118420 -3685> 2013-04-03 10:48:03.602333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140504 v188) v2 from mds.? 10.252.58.31:6800/10908 -3684> 2013-04-03 10:48:03.602347 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140504 v188) v2 -- ?+0 0x2a52600 con 0x2118420 -3683> 2013-04-03 10:48:03.849216 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 12 ==== pg_stats(1 pgs tid 2 v 608) v1 ==== 485+0+0 (3864973743 0 0) 0x33e4b40 con 0x33112c0 -3682> 2013-04-03 10:48:03.849295 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29148 preprocess_query pg_stats(1 pgs tid 2 v 608) v1 from osd.3 10.252.24.88:6804/18376 -3681> 2013-04-03 10:48:03.849321 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29148 prepare_update pg_stats(1 pgs tid 2 v 608) v1 from osd.3 10.252.24.88:6804/18376 -3680> 2013-04-03 10:48:03.849325 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29148 prepare_pg_stats pg_stats(1 pgs tid 2 v 608) v1 from osd.3 -3679> 2013-04-03 10:48:03.849332 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29148 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1188 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -3678> 2013-04-03 10:48:03.899521 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29148 encode_pending v 29149 -3677> 2013-04-03 10:48:03.899569 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427578) queue_proposal bl 654 bytes; ctx = 0x2c3df60 -3676> 2013-04-03 10:48:03.899577 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427578) propose_queued 427579 654 bytes -3675> 2013-04-03 10:48:03.901027 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427578 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3450a00 -3674> 2013-04-03 10:48:04.294139 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 79 ==== paxos(accept lc 427578 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (1786318763 0 0) 0x3450a00 con 0x208a840 -3673> 2013-04-03 10:48:04.295085 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427579 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3450c80 -3672> 2013-04-03 10:48:04.295106 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427579 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3434000 -3671> 2013-04-03 10:48:04.295191 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29148 update_from_paxos applying incremental 29149 -3670> 2013-04-03 10:48:04.295213 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 v29149: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 275B/s wr, 0op/s -3669> 2013-04-03 10:48:04.295232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 send_pg_creates to 0 pgs -3668> 2013-04-03 10:48:04.295238 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 update_logger -3667> 2013-04-03 10:48:04.295249 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 create_pending v 29150 -3666> 2013-04-03 10:48:04.295258 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29149 _updated_stats for osd.3 10.252.24.88:6804/18376 -3665> 2013-04-03 10:48:04.295264 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6804/18376 -- pg_stats_ack(1 pgs tid 2) v1 -- ?+0 0x2224700 con 0x33112c0 -3664> 2013-04-03 10:48:04.295280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 check_osd_map already seen 608 >= 608 -3663> 2013-04-03 10:48:04.295283 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 update_logger -3662> 2013-04-03 10:48:04.295293 7fcdffcdb700 0 log [INF] : pgmap v29149: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 275B/s wr, 0op/s -3661> 2013-04-03 10:48:04.295295 7fcdffcdb700 10 send_log to self -3660> 2013-04-03 10:48:04.295296 7fcdffcdb700 10 log_queue is 1 last_log 226 sent 225 num 1 unsent 1 sending 1 -3659> 2013-04-03 10:48:04.295297 7fcdffcdb700 10 will send 2013-04-03 10:48:04.295294 mon.0 10.252.24.25:6789/0 226 : [INF] pgmap v29149: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 275B/s wr, 0op/s -3658> 2013-04-03 10:48:04.295304 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x33b8200 -3657> 2013-04-03 10:48:04.295354 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos -3656> 2013-04-03 10:48:04.295374 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos version 394135 summary v 394135 -3655> 2013-04-03 10:48:04.295393 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3654> 2013-04-03 10:48:04.295414 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x33b8200 con 0x1bac160 -3653> 2013-04-03 10:48:04.295437 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos -3652> 2013-04-03 10:48:04.295452 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos version 394135 summary v 394135 -3651> 2013-04-03 10:48:04.295460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3650> 2013-04-03 10:48:04.295470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 preprocess_log log(1 entries) v1 from mon.0 -3649> 2013-04-03 10:48:04.295497 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3648> 2013-04-03 10:48:04.295510 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 prepare_log log(1 entries) v1 from mon.0 -3647> 2013-04-03 10:48:04.295520 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 logging 2013-04-03 10:48:04.295294 mon.0 10.252.24.25:6789/0 226 : [INF] pgmap v29149: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 275B/s wr, 0op/s -3646> 2013-04-03 10:48:04.461051 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 80 ==== paxos(lease_ack lc 427579 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1106788737 0 0) 0x3434000 con 0x208a840 -3645> 2013-04-03 10:48:04.461144 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos -3644> 2013-04-03 10:48:04.461164 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394135 update_from_paxos version 394135 summary v 394135 -3643> 2013-04-03 10:48:04.461182 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3642> 2013-04-03 10:48:04.948360 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 81 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 219 v188) v2) to leader v1 ==== 560+0+0 (1403690629 0 0) 0x2cc4580 con 0x208a840 -3641> 2013-04-03 10:48:04.948382 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -3640> 2013-04-03 10:48:04.948393 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2cc4840 from 10.252.24.88:6789/0 -3639> 2013-04-03 10:48:04.948967 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 219 v188) v2 from mds.0 10.252.24.25:6802/3144 -3638> 2013-04-03 10:48:04.948984 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 219 v188) v2 tid 18) v2 -- ?+0 0x33e4b40 con 0x208a840 -3637> 2013-04-03 10:48:05.294347 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394135 encode_pending v394136 -3636> 2013-04-03 10:48:05.294380 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427579) queue_proposal bl 442 bytes; ctx = 0x330b150 -3635> 2013-04-03 10:48:05.294388 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427579) propose_queued 427580 442 bytes -3634> 2013-04-03 10:48:05.295336 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 82 ==== forward(pg_stats(1 pgs tid 54 v 608) v1) to leader v1 ==== 757+0+0 (748337775 0 0) 0x2cc5340 con 0x208a840 -3633> 2013-04-03 10:48:05.295591 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427579 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3450780 -3632> 2013-04-03 10:48:05.295616 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -3631> 2013-04-03 10:48:05.295627 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x33e4b40 from 10.252.24.88:6789/0 -3630> 2013-04-03 10:48:05.295664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 preprocess_query pg_stats(1 pgs tid 54 v 608) v1 from osd.2 10.252.24.30:6801/5656 -3629> 2013-04-03 10:48:05.295688 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 prepare_update pg_stats(1 pgs tid 54 v 608) v1 from osd.2 10.252.24.30:6801/5656 -3628> 2013-04-03 10:48:05.295694 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 prepare_pg_stats pg_stats(1 pgs tid 54 v 608) v1 from osd.2 -3627> 2013-04-03 10:48:05.295700 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29149 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1181 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -3626> 2013-04-03 10:48:05.318822 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 83 ==== paxos(accept lc 427579 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (1161284328 0 0) 0x3450780 con 0x208a840 -3625> 2013-04-03 10:48:05.319134 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427580 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3434000 -3624> 2013-04-03 10:48:05.319152 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427580 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3450a00 -3623> 2013-04-03 10:48:05.319197 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos -3622> 2013-04-03 10:48:05.319232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos version 394136 summary v 394135 -3621> 2013-04-03 10:48:05.319247 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos latest full 393798 -3620> 2013-04-03 10:48:05.319264 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394136 update_from_paxos applying incremental log 394136 2013-04-03 10:48:04.295294 mon.0 10.252.24.25:6789/0 226 : [INF] pgmap v29149: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 275B/s wr, 0op/s -3619> 2013-04-03 10:48:05.319313 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 check_subs -3618> 2013-04-03 10:48:05.319349 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 create_pending v 394137 -3617> 2013-04-03 10:48:05.319372 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394136 _updated_log for mon.0 10.252.24.25:6789/0 -3616> 2013-04-03 10:48:05.319385 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 226) v1 -- ?+0 0x2224c40 con 0x1bac160 -3615> 2013-04-03 10:48:05.319433 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos -3614> 2013-04-03 10:48:05.319449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos version 394136 summary v 394136 -3613> 2013-04-03 10:48:05.319466 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3612> 2013-04-03 10:48:05.319488 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 226) v1 ==== 0+0+0 (0 0 0) 0x2224c40 con 0x1bac160 -3611> 2013-04-03 10:48:05.319496 7fcdffcdb700 10 handle_log_ack log(last 226) v1 -3610> 2013-04-03 10:48:05.319498 7fcdffcdb700 10 logged 2013-04-03 10:48:04.295294 mon.0 10.252.24.25:6789/0 226 : [INF] pgmap v29149: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 275B/s wr, 0op/s -3609> 2013-04-03 10:48:05.342822 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 84 ==== paxos(lease_ack lc 427580 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (4016996270 0 0) 0x3450a00 con 0x208a840 -3608> 2013-04-03 10:48:05.342873 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos -3607> 2013-04-03 10:48:05.342893 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos version 394136 summary v 394136 -3606> 2013-04-03 10:48:05.342908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3605> 2013-04-03 10:48:05.345844 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29149 encode_pending v 29150 -3604> 2013-04-03 10:48:05.345878 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427580) queue_proposal bl 654 bytes; ctx = 0x330b170 -3603> 2013-04-03 10:48:05.345885 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427580) propose_queued 427581 654 bytes -3602> 2013-04-03 10:48:05.346135 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427580 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3450500 -3601> 2013-04-03 10:48:05.355998 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 85 ==== paxos(accept lc 427580 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (2378030439 0 0) 0x3450500 con 0x208a840 -3600> 2013-04-03 10:48:05.356289 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427581 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3450a00 -3599> 2013-04-03 10:48:05.356306 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427581 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3450780 -3598> 2013-04-03 10:48:05.356378 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29149 update_from_paxos applying incremental 29150 -3597> 2013-04-03 10:48:05.356397 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 v29150: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2403B/s wr, 0op/s -3596> 2013-04-03 10:48:05.356416 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 send_pg_creates to 0 pgs -3595> 2013-04-03 10:48:05.356419 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 update_logger -3594> 2013-04-03 10:48:05.356428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 create_pending v 29151 -3593> 2013-04-03 10:48:05.356437 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29150 _updated_stats for osd.2 10.252.24.30:6801/5656 -3592> 2013-04-03 10:48:05.356442 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 54) v1 tid 19) v2 -- ?+0 0x33e4000 con 0x208a840 -3591> 2013-04-03 10:48:05.356453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 check_osd_map already seen 608 >= 608 -3590> 2013-04-03 10:48:05.356455 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 update_logger -3589> 2013-04-03 10:48:05.356465 7fcdffcdb700 0 log [INF] : pgmap v29150: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2403B/s wr, 0op/s -3588> 2013-04-03 10:48:05.356466 7fcdffcdb700 10 send_log to self -3587> 2013-04-03 10:48:05.356467 7fcdffcdb700 10 log_queue is 1 last_log 227 sent 226 num 1 unsent 1 sending 1 -3586> 2013-04-03 10:48:05.356469 7fcdffcdb700 10 will send 2013-04-03 10:48:05.356466 mon.0 10.252.24.25:6789/0 227 : [INF] pgmap v29150: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2403B/s wr, 0op/s -3585> 2013-04-03 10:48:05.356474 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x33b8000 -3584> 2013-04-03 10:48:05.356526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos -3583> 2013-04-03 10:48:05.356544 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos version 394136 summary v 394136 -3582> 2013-04-03 10:48:05.356560 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3581> 2013-04-03 10:48:05.356580 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x33b8000 con 0x1bac160 -3580> 2013-04-03 10:48:05.356613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos -3579> 2013-04-03 10:48:05.356628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos version 394136 summary v 394136 -3578> 2013-04-03 10:48:05.356636 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3577> 2013-04-03 10:48:05.356648 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 preprocess_log log(1 entries) v1 from mon.0 -3576> 2013-04-03 10:48:05.356669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3575> 2013-04-03 10:48:05.356681 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 prepare_log log(1 entries) v1 from mon.0 -3574> 2013-04-03 10:48:05.356690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 logging 2013-04-03 10:48:05.356466 mon.0 10.252.24.25:6789/0 227 : [INF] pgmap v29150: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2403B/s wr, 0op/s -3573> 2013-04-03 10:48:05.368048 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 86 ==== paxos(lease_ack lc 427581 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3584637453 0 0) 0x3450780 con 0x208a840 -3572> 2013-04-03 10:48:05.368093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos -3571> 2013-04-03 10:48:05.368110 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394136 update_from_paxos version 394136 summary v 394136 -3570> 2013-04-03 10:48:05.368125 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3569> 2013-04-03 10:48:05.437334 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 17 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 10 v188) v2 ==== 289+0+0 (401764939 0 0) 0x2f792c0 con 0x3311000 -3568> 2013-04-03 10:48:05.437414 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 10 v188) v2 from mds.? 10.252.24.88:6800/18040 -3567> 2013-04-03 10:48:05.437428 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 10 v188) v2 -- ?+0 0x2cc5340 con 0x3311000 -3566> 2013-04-03 10:48:06.356177 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394136 encode_pending v394137 -3565> 2013-04-03 10:48:06.356205 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427581) queue_proposal bl 443 bytes; ctx = 0x330b120 -3564> 2013-04-03 10:48:06.356215 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427581) propose_queued 427582 443 bytes -3563> 2013-04-03 10:48:06.357353 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427581 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3450280 -3562> 2013-04-03 10:48:06.384237 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 87 ==== paxos(accept lc 427581 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (2287090992 0 0) 0x3450280 con 0x208a840 -3561> 2013-04-03 10:48:06.384640 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427582 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3450780 -3560> 2013-04-03 10:48:06.384661 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427582 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3450500 -3559> 2013-04-03 10:48:06.384703 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos -3558> 2013-04-03 10:48:06.384733 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394136 -3557> 2013-04-03 10:48:06.384747 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos latest full 393798 -3556> 2013-04-03 10:48:06.384764 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394137 update_from_paxos applying incremental log 394137 2013-04-03 10:48:05.356466 mon.0 10.252.24.25:6789/0 227 : [INF] pgmap v29150: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2403B/s wr, 0op/s -3555> 2013-04-03 10:48:06.384814 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 check_subs -3554> 2013-04-03 10:48:06.384850 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 create_pending v 394138 -3553> 2013-04-03 10:48:06.384872 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394137 _updated_log for mon.0 10.252.24.25:6789/0 -3552> 2013-04-03 10:48:06.384885 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 227) v1 -- ?+0 0x2224c40 con 0x1bac160 -3551> 2013-04-03 10:48:06.384937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos -3550> 2013-04-03 10:48:06.384953 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394137 -3549> 2013-04-03 10:48:06.384971 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3548> 2013-04-03 10:48:06.384992 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 227) v1 ==== 0+0+0 (0 0 0) 0x2224c40 con 0x1bac160 -3547> 2013-04-03 10:48:06.385000 7fcdffcdb700 10 handle_log_ack log(last 227) v1 -3546> 2013-04-03 10:48:06.385002 7fcdffcdb700 10 logged 2013-04-03 10:48:05.356466 mon.0 10.252.24.25:6789/0 227 : [INF] pgmap v29150: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2403B/s wr, 0op/s -3545> 2013-04-03 10:48:06.409557 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 88 ==== paxos(lease_ack lc 427582 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1588672801 0 0) 0x3450500 con 0x208a840 -3544> 2013-04-03 10:48:06.409603 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos -3543> 2013-04-03 10:48:06.409620 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394137 -3542> 2013-04-03 10:48:06.409639 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3541> 2013-04-03 10:48:07.601831 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 222 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140505 v188) v2 ==== 289+0+0 (2155753192 0 0) 0x20e4600 con 0x2118420 -3540> 2013-04-03 10:48:07.601911 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140505 v188) v2 from mds.? 10.252.58.31:6800/10908 -3539> 2013-04-03 10:48:07.601924 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140505 v188) v2 -- ?+0 0x2f792c0 con 0x2118420 -3538> 2013-04-03 10:48:08.387817 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29150 check_down_pgs -3537> 2013-04-03 10:48:08.387852 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29150 v29150: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2403B/s wr, 0op/s -3536> 2013-04-03 10:48:08.387881 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -3535> 2013-04-03 10:48:08.387913 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -3534> 2013-04-03 10:48:08.387952 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -3533> 2013-04-03 10:48:08.387983 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos -3532> 2013-04-03 10:48:08.388002 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394137 -3531> 2013-04-03 10:48:08.388012 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394137 log -3530> 2013-04-03 10:48:08.388029 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3529> 2013-04-03 10:48:08.388048 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -3528> 2013-04-03 10:48:08.948556 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 89 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 220 v188) v2) to leader v1 ==== 560+0+0 (558091710 0 0) 0x2ce2b80 con 0x208a840 -3527> 2013-04-03 10:48:08.948585 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -3526> 2013-04-03 10:48:08.948600 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x20e4340 from 10.252.24.88:6789/0 -3525> 2013-04-03 10:48:08.948660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 220 v188) v2 from mds.0 10.252.24.25:6802/3144 -3524> 2013-04-03 10:48:08.948672 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 220 v188) v2 tid 20) v2 -- ?+0 0x33e4b40 con 0x208a840 -3523> 2013-04-03 10:48:09.384743 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427582 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3450000 -3522> 2013-04-03 10:48:09.385550 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 90 ==== paxos(lease_ack lc 427582 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1405818519 0 0) 0x3450000 con 0x208a840 -3521> 2013-04-03 10:48:09.385620 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos -3520> 2013-04-03 10:48:09.385940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394137 -3519> 2013-04-03 10:48:09.385965 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3518> 2013-04-03 10:48:09.437429 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 18 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 11 v188) v2 ==== 289+0+0 (882691382 0 0) 0x346eb00 con 0x3311000 -3517> 2013-04-03 10:48:09.437513 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 11 v188) v2 from mds.? 10.252.24.88:6800/18040 -3516> 2013-04-03 10:48:09.437526 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 11 v188) v2 -- ?+0 0x2ce2b80 con 0x3311000 -3515> 2013-04-03 10:48:09.648131 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 10 ==== pg_stats(1 pgs tid 72 v 608) v1 ==== 477+0+0 (930903481 0 0) 0x33e4480 con 0x3311160 -3514> 2013-04-03 10:48:09.648204 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 preprocess_query pg_stats(1 pgs tid 72 v 608) v1 from osd.1 10.252.24.25:6805/3572 -3513> 2013-04-03 10:48:09.648229 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 prepare_update pg_stats(1 pgs tid 72 v 608) v1 from osd.1 10.252.24.25:6805/3572 -3512> 2013-04-03 10:48:09.648237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 prepare_pg_stats pg_stats(1 pgs tid 72 v 608) v1 from osd.1 -3511> 2013-04-03 10:48:09.648243 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29150 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) -3510> 2013-04-03 10:48:09.698432 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29150 encode_pending v 29151 -3509> 2013-04-03 10:48:09.698473 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427582) queue_proposal bl 646 bytes; ctx = 0x2c3d5f0 -3508> 2013-04-03 10:48:09.698483 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427582) propose_queued 427583 646 bytes -3507> 2013-04-03 10:48:09.699601 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427582 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3476c80 -3506> 2013-04-03 10:48:09.740424 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 91 ==== paxos(accept lc 427582 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (2837925562 0 0) 0x3476c80 con 0x208a840 -3505> 2013-04-03 10:48:09.740779 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427583 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3450000 -3504> 2013-04-03 10:48:09.740802 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427583 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3450500 -3503> 2013-04-03 10:48:09.740901 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29150 update_from_paxos applying incremental 29151 -3502> 2013-04-03 10:48:09.740922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 v29151: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 4944B/s wr, 0op/s -3501> 2013-04-03 10:48:09.740941 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 send_pg_creates to 0 pgs -3500> 2013-04-03 10:48:09.740943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 update_logger -3499> 2013-04-03 10:48:09.740953 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 create_pending v 29152 -3498> 2013-04-03 10:48:09.740962 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29151 _updated_stats for osd.1 10.252.24.25:6805/3572 -3497> 2013-04-03 10:48:09.740968 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(1 pgs tid 72) v1 -- ?+0 0x2224c40 con 0x3311160 -3496> 2013-04-03 10:48:09.740979 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 check_osd_map already seen 608 >= 608 -3495> 2013-04-03 10:48:09.740981 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 update_logger -3494> 2013-04-03 10:48:09.740991 7fcdffcdb700 0 log [INF] : pgmap v29151: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 4944B/s wr, 0op/s -3493> 2013-04-03 10:48:09.740993 7fcdffcdb700 10 send_log to self -3492> 2013-04-03 10:48:09.740993 7fcdffcdb700 10 log_queue is 1 last_log 228 sent 227 num 1 unsent 1 sending 1 -3491> 2013-04-03 10:48:09.740995 7fcdffcdb700 10 will send 2013-04-03 10:48:09.740992 mon.0 10.252.24.25:6789/0 228 : [INF] pgmap v29151: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 4944B/s wr, 0op/s -3490> 2013-04-03 10:48:09.741007 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2545c00 -3489> 2013-04-03 10:48:09.741102 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos -3488> 2013-04-03 10:48:09.741123 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394137 -3487> 2013-04-03 10:48:09.741142 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3486> 2013-04-03 10:48:09.741164 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2545c00 con 0x1bac160 -3485> 2013-04-03 10:48:09.741187 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos -3484> 2013-04-03 10:48:09.741202 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394137 -3483> 2013-04-03 10:48:09.741211 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3482> 2013-04-03 10:48:09.741221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 preprocess_log log(1 entries) v1 from mon.0 -3481> 2013-04-03 10:48:09.741244 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3480> 2013-04-03 10:48:09.741255 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 prepare_log log(1 entries) v1 from mon.0 -3479> 2013-04-03 10:48:09.741265 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 logging 2013-04-03 10:48:09.740992 mon.0 10.252.24.25:6789/0 228 : [INF] pgmap v29151: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 4944B/s wr, 0op/s -3478> 2013-04-03 10:48:09.753117 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 92 ==== paxos(lease_ack lc 427583 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (2037284049 0 0) 0x3450500 con 0x208a840 -3477> 2013-04-03 10:48:09.753159 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos -3476> 2013-04-03 10:48:09.753176 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394137 update_from_paxos version 394137 summary v 394137 -3475> 2013-04-03 10:48:09.753191 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3474> 2013-04-03 10:48:10.740615 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394137 encode_pending v394138 -3473> 2013-04-03 10:48:10.740654 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427583) queue_proposal bl 443 bytes; ctx = 0x330b130 -3472> 2013-04-03 10:48:10.740667 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427583) propose_queued 427584 443 bytes -3471> 2013-04-03 10:48:10.741538 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427583 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3476a00 -3470> 2013-04-03 10:48:10.767797 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 93 ==== paxos(accept lc 427583 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (1765474514 0 0) 0x3476a00 con 0x208a840 -3469> 2013-04-03 10:48:10.768091 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427584 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3450500 -3468> 2013-04-03 10:48:10.768110 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427584 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3476c80 -3467> 2013-04-03 10:48:10.768145 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos -3466> 2013-04-03 10:48:10.768187 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394137 -3465> 2013-04-03 10:48:10.768201 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos latest full 393798 -3464> 2013-04-03 10:48:10.768219 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394138 update_from_paxos applying incremental log 394138 2013-04-03 10:48:09.740992 mon.0 10.252.24.25:6789/0 228 : [INF] pgmap v29151: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 4944B/s wr, 0op/s -3463> 2013-04-03 10:48:10.768267 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 check_subs -3462> 2013-04-03 10:48:10.768305 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 create_pending v 394139 -3461> 2013-04-03 10:48:10.768327 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394138 _updated_log for mon.0 10.252.24.25:6789/0 -3460> 2013-04-03 10:48:10.768340 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 228) v1 -- ?+0 0x22241c0 con 0x1bac160 -3459> 2013-04-03 10:48:10.768393 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos -3458> 2013-04-03 10:48:10.768409 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 -3457> 2013-04-03 10:48:10.768426 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3456> 2013-04-03 10:48:10.768445 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 228) v1 ==== 0+0+0 (0 0 0) 0x22241c0 con 0x1bac160 -3455> 2013-04-03 10:48:10.768452 7fcdffcdb700 10 handle_log_ack log(last 228) v1 -3454> 2013-04-03 10:48:10.768455 7fcdffcdb700 10 logged 2013-04-03 10:48:09.740992 mon.0 10.252.24.25:6789/0 228 : [INF] pgmap v29151: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 4944B/s wr, 0op/s -3453> 2013-04-03 10:48:10.780527 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 94 ==== paxos(lease_ack lc 427584 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (4084458207 0 0) 0x3476c80 con 0x208a840 -3452> 2013-04-03 10:48:10.780610 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos -3451> 2013-04-03 10:48:10.780632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 -3450> 2013-04-03 10:48:10.780649 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3449> 2013-04-03 10:48:11.601946 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 223 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140506 v188) v2 ==== 289+0+0 (3855478639 0 0) 0x20e4b80 con 0x2118420 -3448> 2013-04-03 10:48:11.602024 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140506 v188) v2 from mds.? 10.252.58.31:6800/10908 -3447> 2013-04-03 10:48:11.602041 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140506 v188) v2 -- ?+0 0x346eb00 con 0x2118420 -3446> 2013-04-03 10:48:12.948658 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 95 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 221 v188) v2) to leader v1 ==== 560+0+0 (671749053 0 0) 0x20e4600 con 0x208a840 -3445> 2013-04-03 10:48:12.948681 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -3444> 2013-04-03 10:48:12.948691 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x346fb80 from 10.252.24.88:6789/0 -3443> 2013-04-03 10:48:12.948749 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 221 v188) v2 from mds.0 10.252.24.25:6802/3144 -3442> 2013-04-03 10:48:12.948761 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 221 v188) v2 tid 21) v2 -- ?+0 0x33e4480 con 0x208a840 -3441> 2013-04-03 10:48:13.388176 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29151 check_down_pgs -3440> 2013-04-03 10:48:13.388208 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29151 v29151: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 4944B/s wr, 0op/s -3439> 2013-04-03 10:48:13.388238 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -3438> 2013-04-03 10:48:13.388261 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -3437> 2013-04-03 10:48:13.388299 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -3436> 2013-04-03 10:48:13.388332 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos -3435> 2013-04-03 10:48:13.388351 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 -3434> 2013-04-03 10:48:13.388360 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394138 log -3433> 2013-04-03 10:48:13.388377 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3432> 2013-04-03 10:48:13.388396 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -3431> 2013-04-03 10:48:13.437546 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 19 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 12 v188) v2 ==== 289+0+0 (3700814149 0 0) 0x346e840 con 0x3311000 -3430> 2013-04-03 10:48:13.437630 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 12 v188) v2 from mds.? 10.252.24.88:6800/18040 -3429> 2013-04-03 10:48:13.437644 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 12 v188) v2 -- ?+0 0x20e4600 con 0x3311000 -3428> 2013-04-03 10:48:13.768188 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427584 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3476780 -3427> 2013-04-03 10:48:13.768962 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 96 ==== paxos(lease_ack lc 427584 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (2732536163 0 0) 0x3476780 con 0x208a840 -3426> 2013-04-03 10:48:13.769037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos -3425> 2013-04-03 10:48:13.769057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 -3424> 2013-04-03 10:48:13.769075 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3423> 2013-04-03 10:48:15.602076 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 224 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140507 v188) v2 ==== 289+0+0 (3332591634 0 0) 0x20e3b00 con 0x2118420 -3422> 2013-04-03 10:48:15.602161 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140507 v188) v2 from mds.? 10.252.58.31:6800/10908 -3421> 2013-04-03 10:48:15.602175 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140507 v188) v2 -- ?+0 0x346e840 con 0x2118420 -3420> 2013-04-03 10:48:16.768254 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427584 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3476500 -3419> 2013-04-03 10:48:16.769029 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 97 ==== paxos(lease_ack lc 427584 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (4234835609 0 0) 0x3476500 con 0x208a840 -3418> 2013-04-03 10:48:16.769107 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos -3417> 2013-04-03 10:48:16.769128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 -3416> 2013-04-03 10:48:16.769151 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3415> 2013-04-03 10:48:16.948776 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 98 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 222 v188) v2) to leader v1 ==== 560+0+0 (869298616 0 0) 0x20e4b80 con 0x208a840 -3414> 2013-04-03 10:48:16.948798 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -3413> 2013-04-03 10:48:16.948808 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x346f8c0 from 10.252.24.88:6789/0 -3412> 2013-04-03 10:48:16.948865 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 222 v188) v2 from mds.0 10.252.24.25:6802/3144 -3411> 2013-04-03 10:48:16.948877 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 222 v188) v2 tid 22) v2 -- ?+0 0x33e4b40 con 0x208a840 -3410> 2013-04-03 10:48:17.437687 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 20 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 13 v188) v2 ==== 289+0+0 (4294659640 0 0) 0x346e580 con 0x3311000 -3409> 2013-04-03 10:48:17.437774 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 13 v188) v2 from mds.? 10.252.24.88:6800/18040 -3408> 2013-04-03 10:48:17.437787 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 13 v188) v2 -- ?+0 0x20e4b80 con 0x3311000 -3407> 2013-04-03 10:48:18.388521 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29151 v29151: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 4944B/s wr, 0op/s -3406> 2013-04-03 10:48:18.388564 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -3405> 2013-04-03 10:48:18.388587 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -3404> 2013-04-03 10:48:18.388632 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -3403> 2013-04-03 10:48:18.388661 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos -3402> 2013-04-03 10:48:18.388689 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 -3401> 2013-04-03 10:48:18.388697 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394138 log -3400> 2013-04-03 10:48:18.388714 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3399> 2013-04-03 10:48:18.388733 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -3398> 2013-04-03 10:48:19.602213 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 225 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140508 v188) v2 ==== 289+0+0 (782920801 0 0) 0x2ef9b00 con 0x2118420 -3397> 2013-04-03 10:48:19.602298 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140508 v188) v2 from mds.? 10.252.58.31:6800/10908 -3396> 2013-04-03 10:48:19.602312 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140508 v188) v2 -- ?+0 0x346e580 con 0x2118420 -3395> 2013-04-03 10:48:19.768318 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427584 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3476280 -3394> 2013-04-03 10:48:19.769153 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 99 ==== paxos(lease_ack lc 427584 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (4044240888 0 0) 0x3476280 con 0x208a840 -3393> 2013-04-03 10:48:19.769232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos -3392> 2013-04-03 10:48:19.769252 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 -3391> 2013-04-03 10:48:19.769270 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3390> 2013-04-03 10:48:20.297008 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 100 ==== forward(pg_stats(1 pgs tid 55 v 608) v1) to leader v1 ==== 757+0+0 (174565323 0 0) 0x20e3b00 con 0x208a840 -3389> 2013-04-03 10:48:20.297037 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -3388> 2013-04-03 10:48:20.297047 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x33e4b40 from 10.252.24.88:6789/0 -3387> 2013-04-03 10:48:20.297104 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 preprocess_query pg_stats(1 pgs tid 55 v 608) v1 from osd.2 10.252.24.30:6801/5656 -3386> 2013-04-03 10:48:20.297128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 prepare_update pg_stats(1 pgs tid 55 v 608) v1 from osd.2 10.252.24.30:6801/5656 -3385> 2013-04-03 10:48:20.297134 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 prepare_pg_stats pg_stats(1 pgs tid 55 v 608) v1 from osd.2 -3384> 2013-04-03 10:48:20.297140 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29151 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -3383> 2013-04-03 10:48:20.347312 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29151 encode_pending v 29152 -3382> 2013-04-03 10:48:20.347357 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427584) queue_proposal bl 654 bytes; ctx = 0x330b0e0 -3381> 2013-04-03 10:48:20.347366 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427584) propose_queued 427585 654 bytes -3380> 2013-04-03 10:48:20.348481 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427584 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3476000 -3379> 2013-04-03 10:48:20.394102 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 101 ==== paxos(accept lc 427584 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (1761826517 0 0) 0x3476000 con 0x208a840 -3378> 2013-04-03 10:48:20.394393 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427585 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3476280 -3377> 2013-04-03 10:48:20.394413 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427585 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3476500 -3376> 2013-04-03 10:48:20.394487 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29151 update_from_paxos applying incremental 29152 -3375> 2013-04-03 10:48:20.394508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 v29152: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -3374> 2013-04-03 10:48:20.394528 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 send_pg_creates to 0 pgs -3373> 2013-04-03 10:48:20.394530 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 update_logger -3372> 2013-04-03 10:48:20.394540 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 create_pending v 29153 -3371> 2013-04-03 10:48:20.394549 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29152 _updated_stats for osd.2 10.252.24.30:6801/5656 -3370> 2013-04-03 10:48:20.394555 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 55) v1 tid 23) v2 -- ?+0 0x33e4480 con 0x208a840 -3369> 2013-04-03 10:48:20.394566 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 check_osd_map already seen 608 >= 608 -3368> 2013-04-03 10:48:20.394568 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 update_logger -3367> 2013-04-03 10:48:20.394578 7fcdffcdb700 0 log [INF] : pgmap v29152: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -3366> 2013-04-03 10:48:20.394580 7fcdffcdb700 10 send_log to self -3365> 2013-04-03 10:48:20.394581 7fcdffcdb700 10 log_queue is 1 last_log 229 sent 228 num 1 unsent 1 sending 1 -3364> 2013-04-03 10:48:20.394590 7fcdffcdb700 10 will send 2013-04-03 10:48:20.394580 mon.0 10.252.24.25:6789/0 229 : [INF] pgmap v29152: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -3363> 2013-04-03 10:48:20.394598 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2545c00 -3362> 2013-04-03 10:48:20.394652 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos -3361> 2013-04-03 10:48:20.394672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 -3360> 2013-04-03 10:48:20.394690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3359> 2013-04-03 10:48:20.394711 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2545c00 con 0x1bac160 -3358> 2013-04-03 10:48:20.394733 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos -3357> 2013-04-03 10:48:20.394748 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 -3356> 2013-04-03 10:48:20.394757 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3355> 2013-04-03 10:48:20.394768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 preprocess_log log(1 entries) v1 from mon.0 -3354> 2013-04-03 10:48:20.394790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3353> 2013-04-03 10:48:20.394801 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 prepare_log log(1 entries) v1 from mon.0 -3352> 2013-04-03 10:48:20.394810 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 logging 2013-04-03 10:48:20.394580 mon.0 10.252.24.25:6789/0 229 : [INF] pgmap v29152: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -3351> 2013-04-03 10:48:20.407146 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 102 ==== paxos(lease_ack lc 427585 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3979233707 0 0) 0x3476500 con 0x208a840 -3350> 2013-04-03 10:48:20.407229 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos -3349> 2013-04-03 10:48:20.407249 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394138 update_from_paxos version 394138 summary v 394138 -3348> 2013-04-03 10:48:20.407266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3347> 2013-04-03 10:48:20.948905 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 103 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 223 v188) v2) to leader v1 ==== 560+0+0 (2959245990 0 0) 0x346f600 con 0x208a840 -3346> 2013-04-03 10:48:20.948934 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -3345> 2013-04-03 10:48:20.948947 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x346f340 from 10.252.24.88:6789/0 -3344> 2013-04-03 10:48:20.949002 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 223 v188) v2 from mds.0 10.252.24.25:6802/3144 -3343> 2013-04-03 10:48:20.949013 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 223 v188) v2 tid 24) v2 -- ?+0 0x33e4b40 con 0x208a840 -3342> 2013-04-03 10:48:21.394291 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394138 encode_pending v394139 -3341> 2013-04-03 10:48:21.394320 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427585) queue_proposal bl 443 bytes; ctx = 0x330b100 -3340> 2013-04-03 10:48:21.394328 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427585) propose_queued 427586 443 bytes -3339> 2013-04-03 10:48:21.395284 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427585 fc 0 pn 17700 opn 0) v3 -- ?+0 0x348cc80 -3338> 2013-04-03 10:48:21.422615 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 104 ==== paxos(accept lc 427585 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (3589082769 0 0) 0x348cc80 con 0x208a840 -3337> 2013-04-03 10:48:21.423327 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427586 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3476500 -3336> 2013-04-03 10:48:21.423348 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427586 fc 427522 pn 0 opn 0) v3 -- ?+0 0x3476000 -3335> 2013-04-03 10:48:21.423387 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos -3334> 2013-04-03 10:48:21.423426 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394138 -3333> 2013-04-03 10:48:21.423441 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos latest full 393798 -3332> 2013-04-03 10:48:21.423458 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394139 update_from_paxos applying incremental log 394139 2013-04-03 10:48:20.394580 mon.0 10.252.24.25:6789/0 229 : [INF] pgmap v29152: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -3331> 2013-04-03 10:48:21.423508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 check_subs -3330> 2013-04-03 10:48:21.423544 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 create_pending v 394140 -3329> 2013-04-03 10:48:21.423566 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394139 _updated_log for mon.0 10.252.24.25:6789/0 -3328> 2013-04-03 10:48:21.423579 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 229) v1 -- ?+0 0x2224000 con 0x1bac160 -3327> 2013-04-03 10:48:21.423633 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos -3326> 2013-04-03 10:48:21.423649 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 -3325> 2013-04-03 10:48:21.423666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3324> 2013-04-03 10:48:21.423687 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 229) v1 ==== 0+0+0 (0 0 0) 0x2224000 con 0x1bac160 -3323> 2013-04-03 10:48:21.423695 7fcdffcdb700 10 handle_log_ack log(last 229) v1 -3322> 2013-04-03 10:48:21.423699 7fcdffcdb700 10 logged 2013-04-03 10:48:20.394580 mon.0 10.252.24.25:6789/0 229 : [INF] pgmap v29152: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -3321> 2013-04-03 10:48:21.434311 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 105 ==== paxos(lease_ack lc 427586 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (1333290812 0 0) 0x3476000 con 0x208a840 -3320> 2013-04-03 10:48:21.434355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos -3319> 2013-04-03 10:48:21.434373 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 -3318> 2013-04-03 10:48:21.434388 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3317> 2013-04-03 10:48:21.437677 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 21 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 14 v188) v2 ==== 289+0+0 (2588446655 0 0) 0x346e2c0 con 0x3311000 -3316> 2013-04-03 10:48:21.437724 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 14 v188) v2 from mds.? 10.252.24.88:6800/18040 -3315> 2013-04-03 10:48:21.437735 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 14 v188) v2 -- ?+0 0x346f600 con 0x3311000 -3314> 2013-04-03 10:48:23.388855 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29152 check_down_pgs -3313> 2013-04-03 10:48:23.388887 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29152 v29152: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -3312> 2013-04-03 10:48:23.388917 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -3311> 2013-04-03 10:48:23.388940 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -3310> 2013-04-03 10:48:23.388980 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -3309> 2013-04-03 10:48:23.389009 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos -3308> 2013-04-03 10:48:23.389032 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 -3307> 2013-04-03 10:48:23.389040 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394139 log -3306> 2013-04-03 10:48:23.389057 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3305> 2013-04-03 10:48:23.389076 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -3304> 2013-04-03 10:48:23.602348 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 226 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140509 v188) v2 ==== 289+0+0 (231002908 0 0) 0x2efa340 con 0x2118420 -3303> 2013-04-03 10:48:23.602432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140509 v188) v2 from mds.? 10.252.58.31:6800/10908 -3302> 2013-04-03 10:48:23.602446 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140509 v188) v2 -- ?+0 0x346e2c0 con 0x2118420 -3301> 2013-04-03 10:48:24.423419 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427586 fc 427522 pn 0 opn 0) v3 -- ?+0 0x348ca00 -3300> 2013-04-03 10:48:24.424199 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 106 ==== paxos(lease_ack lc 427586 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (3240937170 0 0) 0x348ca00 con 0x208a840 -3299> 2013-04-03 10:48:24.424277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos -3298> 2013-04-03 10:48:24.424298 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 -3297> 2013-04-03 10:48:24.424315 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3296> 2013-04-03 10:48:24.948921 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 107 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 224 v188) v2) to leader v1 ==== 560+0+0 (3297244644 0 0) 0x20e3b00 con 0x208a840 -3295> 2013-04-03 10:48:24.948943 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -3294> 2013-04-03 10:48:24.948953 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x346f080 from 10.252.24.88:6789/0 -3293> 2013-04-03 10:48:24.949011 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 224 v188) v2 from mds.0 10.252.24.25:6802/3144 -3292> 2013-04-03 10:48:24.949026 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 224 v188) v2 tid 25) v2 -- ?+0 0x33e4480 con 0x208a840 -3291> 2013-04-03 10:48:25.437854 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 22 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 15 v188) v2 ==== 289+0+0 (3106287810 0 0) 0x346e000 con 0x3311000 -3290> 2013-04-03 10:48:25.437937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 15 v188) v2 from mds.? 10.252.24.88:6800/18040 -3289> 2013-04-03 10:48:25.437951 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 15 v188) v2 -- ?+0 0x20e3b00 con 0x3311000 -3288> 2013-04-03 10:48:27.423484 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427586 fc 427522 pn 0 opn 0) v3 -- ?+0 0x348c780 -3287> 2013-04-03 10:48:27.424282 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 108 ==== paxos(lease_ack lc 427586 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (4294026168 0 0) 0x348c780 con 0x208a840 -3286> 2013-04-03 10:48:27.424360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos -3285> 2013-04-03 10:48:27.424380 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 -3284> 2013-04-03 10:48:27.424398 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3283> 2013-04-03 10:48:27.602465 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 227 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140510 v188) v2 ==== 289+0+0 (1752634011 0 0) 0x2efa600 con 0x2118420 -3282> 2013-04-03 10:48:27.602548 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140510 v188) v2 from mds.? 10.252.58.31:6800/10908 -3281> 2013-04-03 10:48:27.602562 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140510 v188) v2 -- ?+0 0x346e000 con 0x2118420 -3280> 2013-04-03 10:48:28.389199 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29152 v29152: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -3279> 2013-04-03 10:48:28.389241 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -3278> 2013-04-03 10:48:28.389264 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -3277> 2013-04-03 10:48:28.389307 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -3276> 2013-04-03 10:48:28.389347 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos -3275> 2013-04-03 10:48:28.389366 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 -3274> 2013-04-03 10:48:28.389375 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394139 log -3273> 2013-04-03 10:48:28.389393 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3272> 2013-04-03 10:48:28.389411 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -3271> 2013-04-03 10:48:28.852101 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 13 ==== pg_stats(1 pgs tid 3 v 608) v1 ==== 485+0+0 (169016667 0 0) 0x33e4900 con 0x33112c0 -3270> 2013-04-03 10:48:28.852183 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 preprocess_query pg_stats(1 pgs tid 3 v 608) v1 from osd.3 10.252.24.88:6804/18376 -3269> 2013-04-03 10:48:28.852209 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 prepare_update pg_stats(1 pgs tid 3 v 608) v1 from osd.3 10.252.24.88:6804/18376 -3268> 2013-04-03 10:48:28.852214 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 prepare_pg_stats pg_stats(1 pgs tid 3 v 608) v1 from osd.3 -3267> 2013-04-03 10:48:28.852221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29152 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -3266> 2013-04-03 10:48:28.902381 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29152 encode_pending v 29153 -3265> 2013-04-03 10:48:28.902423 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427586) queue_proposal bl 654 bytes; ctx = 0x2c3de10 -3264> 2013-04-03 10:48:28.902433 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427522..427586) propose_queued 427587 654 bytes -3263> 2013-04-03 10:48:28.903301 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427586 fc 0 pn 17700 opn 0) v3 -- ?+0 0x348c500 -3262> 2013-04-03 10:48:28.932497 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 109 ==== paxos(accept lc 427586 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (3812306712 0 0) 0x348c500 con 0x208a840 -3261> 2013-04-03 10:48:28.942330 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427587 fc 0 pn 17700 opn 0) v3 -- ?+0 0x348c780 -3260> 2013-04-03 10:48:28.942370 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427587 fc 427522 pn 0 opn 0) v3 -- ?+0 0x348ca00 -3259> 2013-04-03 10:48:28.942515 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29152 update_from_paxos applying incremental 29153 -3258> 2013-04-03 10:48:28.942536 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29153 v29153: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 906B/s wr, 0op/s -3257> 2013-04-03 10:48:28.942555 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29153 send_pg_creates to 0 pgs -3256> 2013-04-03 10:48:28.942557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29153 update_logger -3255> 2013-04-03 10:48:28.942567 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29153 create_pending v 29154 -3254> 2013-04-03 10:48:28.942576 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29153 _updated_stats for osd.3 10.252.24.88:6804/18376 -3253> 2013-04-03 10:48:28.942582 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6804/18376 -- pg_stats_ack(1 pgs tid 3) v1 -- ?+0 0x2224000 con 0x33112c0 -3252> 2013-04-03 10:48:28.942596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29153 check_osd_map already seen 608 >= 608 -3251> 2013-04-03 10:48:28.942599 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29153 update_logger -3250> 2013-04-03 10:48:28.942609 7fcdffcdb700 0 log [INF] : pgmap v29153: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 906B/s wr, 0op/s -3249> 2013-04-03 10:48:28.942611 7fcdffcdb700 10 send_log to self -3248> 2013-04-03 10:48:28.942612 7fcdffcdb700 10 log_queue is 1 last_log 230 sent 229 num 1 unsent 1 sending 1 -3247> 2013-04-03 10:48:28.942614 7fcdffcdb700 10 will send 2013-04-03 10:48:28.942611 mon.0 10.252.24.25:6789/0 230 : [INF] pgmap v29153: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 906B/s wr, 0op/s -3246> 2013-04-03 10:48:28.942620 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24eb600 -3245> 2013-04-03 10:48:28.942672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos -3244> 2013-04-03 10:48:28.942691 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 -3243> 2013-04-03 10:48:28.942708 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3242> 2013-04-03 10:48:28.942730 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24eb600 con 0x1bac160 -3241> 2013-04-03 10:48:28.942752 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos -3240> 2013-04-03 10:48:28.942768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 -3239> 2013-04-03 10:48:28.942776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3238> 2013-04-03 10:48:28.942787 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 preprocess_log log(1 entries) v1 from mon.0 -3237> 2013-04-03 10:48:28.942811 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3236> 2013-04-03 10:48:28.942822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 prepare_log log(1 entries) v1 from mon.0 -3235> 2013-04-03 10:48:28.942832 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 logging 2013-04-03 10:48:28.942611 mon.0 10.252.24.25:6789/0 230 : [INF] pgmap v29153: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 906B/s wr, 0op/s -3234> 2013-04-03 10:48:28.952554 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 110 ==== paxos(lease_ack lc 427587 fc 427522 pn 0 opn 0) v3 ==== 80+0+0 (789043393 0 0) 0x348ca00 con 0x208a840 -3233> 2013-04-03 10:48:28.952596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos -3232> 2013-04-03 10:48:28.952613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394139 update_from_paxos version 394139 summary v 394139 -3231> 2013-04-03 10:48:28.952628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -3230> 2013-04-03 10:48:28.952822 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 111 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 225 v188) v2) to leader v1 ==== 560+0+0 (1441184131 0 0) 0x2efa340 con 0x208a840 -3229> 2013-04-03 10:48:28.952833 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -3228> 2013-04-03 10:48:28.952840 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x346edc0 from 10.252.24.88:6789/0 -3227> 2013-04-03 10:48:28.952866 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 225 v188) v2 from mds.0 10.252.24.25:6802/3144 -3226> 2013-04-03 10:48:28.952876 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 225 v188) v2 tid 26) v2 -- ?+0 0x33e4d80 con 0x208a840 -3225> 2013-04-03 10:48:29.108193 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427522..427587) queue_proposal bl 1707 bytes; ctx = 0x330b0f0 -3224> 2013-04-03 10:48:29.163977 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 112 ==== mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 ==== 174+0+0 (424405211 0 0) 0x2efa600 con 0x208a840 -3223> 2013-04-03 10:48:29.164008 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -3222> 2013-04-03 10:48:29.164015 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_sync_start mon_sync( start flags( reply-to(mon.1 10.252.24.30:6789/0) ) ) v1 -3221> 2013-04-03 10:48:29.164025 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_start set provider to mon.2 10.252.24.88:6789/0 -3220> 2013-04-03 10:48:29.164031 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( start_reply flags( reply-to(mon.2 10.252.24.88:6789/0) ) ) v1 -- ?+0 0x2efa340 -3219> 2013-04-03 10:48:29.437931 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 23 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 16 v188) v2 ==== 289+0+0 (1921358746 0 0) 0x20e3b00 con 0x3311000 -3218> 2013-04-03 10:48:29.438012 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 16 v188) v2 from mds.? 10.252.24.88:6800/18040 -3217> 2013-04-03 10:48:29.438027 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 16 v188) v2 -- ?+0 0x2efa600 con 0x3311000 -3216> 2013-04-03 10:48:29.650418 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 11 ==== pg_stats(1 pgs tid 73 v 608) v1 ==== 477+0+0 (2543049462 0 0) 0x349e000 con 0x3311160 -3215> 2013-04-03 10:48:29.650491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29153 preprocess_query pg_stats(1 pgs tid 73 v 608) v1 from osd.1 10.252.24.25:6805/3572 -3214> 2013-04-03 10:48:29.650519 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29153 prepare_update pg_stats(1 pgs tid 73 v 608) v1 from osd.1 10.252.24.25:6805/3572 -3213> 2013-04-03 10:48:29.650525 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29153 prepare_pg_stats pg_stats(1 pgs tid 73 v 608) v1 from osd.1 -3212> 2013-04-03 10:48:29.650532 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29153 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) -3211> 2013-04-03 10:48:29.932704 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29153 encode_pending v 29154 -3210> 2013-04-03 10:48:29.932749 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427522..427587) queue_proposal bl 646 bytes; ctx = 0x330b140 -3209> 2013-04-03 10:48:29.932758 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427522..427587) propose_queued 427588 1707 bytes -3208> 2013-04-03 10:48:29.933721 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427587 fc 0 pn 17700 opn 0) v3 -- ?+0 0x348c280 -3207> 2013-04-03 10:48:29.933813 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394139 encode_pending v394140 -3206> 2013-04-03 10:48:29.933847 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427522..427587) queue_proposal bl 442 bytes; ctx = 0x330b0b0 -3205> 2013-04-03 10:48:29.933853 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427522..427587) propose_new_value not active; proposal queued -3204> 2013-04-03 10:48:29.966314 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 113 ==== paxos(accept lc 427587 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (2501663427 0 0) 0x348c280 con 0x208a840 -3203> 2013-04-03 10:48:29.971265 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427588 fc 0 pn 17700 opn 0) v3 -- ?+0 0x348ca00 -3202> 2013-04-03 10:48:29.971289 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427588 fc 427522 pn 0 opn 0) v3 -- ?+0 0x348c500 -3201> 2013-04-03 10:48:29.971346 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427577..427588) propose_queued 427589 646 bytes -3200> 2013-04-03 10:48:29.971538 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427588 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3476000 -3199> 2013-04-03 10:48:29.999512 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 114 ==== paxos(lease_ack lc 427588 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1359556720 0 0) 0x3476000 con 0x208a840 -3198> 2013-04-03 10:48:30.012395 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 115 ==== paxos(accept lc 427588 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (3629454044 0 0) 0x348c500 con 0x208a840 -3197> 2013-04-03 10:48:30.013203 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427589 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3476000 -3196> 2013-04-03 10:48:30.013235 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427589 fc 427577 pn 0 opn 0) v3 -- ?+0 0x348c280 -3195> 2013-04-03 10:48:30.013339 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29153 update_from_paxos applying incremental 29154 -3194> 2013-04-03 10:48:30.013361 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 v29154: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s -3193> 2013-04-03 10:48:30.013381 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 send_pg_creates to 0 pgs -3192> 2013-04-03 10:48:30.013384 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 update_logger -3191> 2013-04-03 10:48:30.013395 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 create_pending v 29155 -3190> 2013-04-03 10:48:30.013404 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 _updated_stats for osd.1 10.252.24.25:6805/3572 -3189> 2013-04-03 10:48:30.013410 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(1 pgs tid 73) v1 -- ?+0 0x2e8b540 con 0x3311160 -3188> 2013-04-03 10:48:30.013422 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 check_osd_map already seen 608 >= 608 -3187> 2013-04-03 10:48:30.013425 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 update_logger -3186> 2013-04-03 10:48:30.013435 7fcdffcdb700 0 log [INF] : pgmap v29154: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s -3185> 2013-04-03 10:48:30.013437 7fcdffcdb700 10 send_log to self -3184> 2013-04-03 10:48:30.013437 7fcdffcdb700 10 log_queue is 2 last_log 231 sent 230 num 2 unsent 1 sending 1 -3183> 2013-04-03 10:48:30.013439 7fcdffcdb700 10 will send 2013-04-03 10:48:28.942611 mon.0 10.252.24.25:6789/0 230 : [INF] pgmap v29153: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 906B/s wr, 0op/s -3182> 2013-04-03 10:48:30.013446 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x287c200 -3181> 2013-04-03 10:48:30.013471 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427577..427589) propose_queued 427590 442 bytes -3180> 2013-04-03 10:48:30.013700 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427589 fc 0 pn 17700 opn 0) v3 -- ?+0 0x348cc80 -3179> 2013-04-03 10:48:30.013737 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x287c200 con 0x1bac160 -3178> 2013-04-03 10:48:30.035883 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 116 ==== paxos(lease_ack lc 427589 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2466732355 0 0) 0x348cc80 con 0x208a840 -3177> 2013-04-03 10:48:30.048938 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 117 ==== paxos(accept lc 427589 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (4217194886 0 0) 0x348c280 con 0x208a840 -3176> 2013-04-03 10:48:30.049264 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427590 fc 0 pn 17700 opn 0) v3 -- ?+0 0x348cc80 -3175> 2013-04-03 10:48:30.049286 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427590 fc 427577 pn 0 opn 0) v3 -- ?+0 0x348c500 -3174> 2013-04-03 10:48:30.049326 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos -3173> 2013-04-03 10:48:30.049365 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos version 394140 summary v 394139 -3172> 2013-04-03 10:48:30.049380 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos latest full 393798 -3171> 2013-04-03 10:48:30.049398 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos applying incremental log 394140 2013-04-03 10:48:28.942611 mon.0 10.252.24.25:6789/0 230 : [INF] pgmap v29153: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 906B/s wr, 0op/s -3170> 2013-04-03 10:48:30.049449 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 check_subs -3169> 2013-04-03 10:48:30.049486 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 create_pending v 394141 -3168> 2013-04-03 10:48:30.049509 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 _updated_log for mon.0 10.252.24.25:6789/0 -3167> 2013-04-03 10:48:30.049525 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 230) v1 -- ?+0 0x28a5c40 con 0x1bac160 -3166> 2013-04-03 10:48:30.049546 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos -3165> 2013-04-03 10:48:30.049561 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos version 394140 summary v 394140 -3164> 2013-04-03 10:48:30.049571 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3163> 2013-04-03 10:48:30.049583 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 preprocess_log log(1 entries) v1 from mon.0 -3162> 2013-04-03 10:48:30.049601 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 nothing new -3161> 2013-04-03 10:48:30.049661 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos -3160> 2013-04-03 10:48:30.049677 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos version 394140 summary v 394140 -3159> 2013-04-03 10:48:30.049696 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3158> 2013-04-03 10:48:30.049719 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 230) v1 ==== 0+0+0 (0 0 0) 0x28a5c40 con 0x1bac160 -3157> 2013-04-03 10:48:30.049728 7fcdffcdb700 10 handle_log_ack log(last 230) v1 -3156> 2013-04-03 10:48:30.049731 7fcdffcdb700 10 logged 2013-04-03 10:48:28.942611 mon.0 10.252.24.25:6789/0 230 : [INF] pgmap v29153: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 906B/s wr, 0op/s -3155> 2013-04-03 10:48:30.060651 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 118 ==== paxos(lease_ack lc 427590 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (83271934 0 0) 0x348c500 con 0x208a840 -3154> 2013-04-03 10:48:30.060740 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos -3153> 2013-04-03 10:48:30.060765 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos version 394140 summary v 394140 -3152> 2013-04-03 10:48:30.060784 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3151> 2013-04-03 10:48:30.297831 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 119 ==== forward(pg_stats(1 pgs tid 56 v 608) v1) to leader v1 ==== 757+0+0 (2854112131 0 0) 0x34ad600 con 0x208a840 -3150> 2013-04-03 10:48:30.297851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -3149> 2013-04-03 10:48:30.297862 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x33e4d80 from 10.252.24.88:6789/0 -3148> 2013-04-03 10:48:30.297917 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 preprocess_query pg_stats(1 pgs tid 56 v 608) v1 from osd.2 10.252.24.30:6801/5656 -3147> 2013-04-03 10:48:30.297941 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 prepare_update pg_stats(1 pgs tid 56 v 608) v1 from osd.2 10.252.24.30:6801/5656 -3146> 2013-04-03 10:48:30.297948 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 prepare_pg_stats pg_stats(1 pgs tid 56 v 608) v1 from osd.2 -3145> 2013-04-03 10:48:30.297955 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -3144> 2013-04-03 10:48:31.049134 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 encode_pending v 29155 -3143> 2013-04-03 10:48:31.049187 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427577..427590) queue_proposal bl 654 bytes; ctx = 0x330b160 -3142> 2013-04-03 10:48:31.049198 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427577..427590) propose_queued 427591 654 bytes -3141> 2013-04-03 10:48:31.050483 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427590 fc 0 pn 17700 opn 0) v3 -- ?+0 0x348c000 -3140> 2013-04-03 10:48:31.076269 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 120 ==== paxos(accept lc 427590 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (2959815243 0 0) 0x348c000 con 0x208a840 -3139> 2013-04-03 10:48:31.076697 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427591 fc 0 pn 17700 opn 0) v3 -- ?+0 0x348c500 -3138> 2013-04-03 10:48:31.076719 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427591 fc 427577 pn 0 opn 0) v3 -- ?+0 0x348c280 -3137> 2013-04-03 10:48:31.076808 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29154 update_from_paxos applying incremental 29155 -3136> 2013-04-03 10:48:31.076829 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 v29155: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5723B/s wr, 0op/s -3135> 2013-04-03 10:48:31.076848 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 send_pg_creates to 0 pgs -3134> 2013-04-03 10:48:31.076851 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 update_logger -3133> 2013-04-03 10:48:31.076865 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 create_pending v 29156 -3132> 2013-04-03 10:48:31.076874 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 _updated_stats for osd.2 10.252.24.30:6801/5656 -3131> 2013-04-03 10:48:31.076880 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 56) v1 tid 27) v2 -- ?+0 0x33e4480 con 0x208a840 -3130> 2013-04-03 10:48:31.076892 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 check_osd_map already seen 608 >= 608 -3129> 2013-04-03 10:48:31.076895 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 update_logger -3128> 2013-04-03 10:48:31.076905 7fcdffcdb700 0 log [INF] : pgmap v29155: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5723B/s wr, 0op/s -3127> 2013-04-03 10:48:31.076907 7fcdffcdb700 10 send_log to self -3126> 2013-04-03 10:48:31.076908 7fcdffcdb700 10 log_queue is 2 last_log 232 sent 230 num 2 unsent 2 sending 2 -3125> 2013-04-03 10:48:31.076910 7fcdffcdb700 10 will send 2013-04-03 10:48:30.013436 mon.0 10.252.24.25:6789/0 231 : [INF] pgmap v29154: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s -3124> 2013-04-03 10:48:31.076921 7fcdffcdb700 10 will send 2013-04-03 10:48:31.076907 mon.0 10.252.24.25:6789/0 232 : [INF] pgmap v29155: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5723B/s wr, 0op/s -3123> 2013-04-03 10:48:31.076926 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x287c200 -3122> 2013-04-03 10:48:31.077281 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos -3121> 2013-04-03 10:48:31.077304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos version 394140 summary v 394140 -3120> 2013-04-03 10:48:31.077322 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3119> 2013-04-03 10:48:31.077343 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x287c200 con 0x1bac160 -3118> 2013-04-03 10:48:31.077366 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos -3117> 2013-04-03 10:48:31.077381 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos version 394140 summary v 394140 -3116> 2013-04-03 10:48:31.077391 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -3115> 2013-04-03 10:48:31.077402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 preprocess_log log(2 entries) v1 from mon.0 -3114> 2013-04-03 10:48:31.077434 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -3113> 2013-04-03 10:48:31.077447 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 prepare_log log(2 entries) v1 from mon.0 -3112> 2013-04-03 10:48:31.077457 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 logging 2013-04-03 10:48:30.013436 mon.0 10.252.24.25:6789/0 231 : [INF] pgmap v29154: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s -3111> 2013-04-03 10:48:31.077480 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 logging 2013-04-03 10:48:31.076907 mon.0 10.252.24.25:6789/0 232 : [INF] pgmap v29155: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5723B/s wr, 0op/s -3110> 2013-04-03 10:48:31.087948 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 121 ==== paxos(lease_ack lc 427591 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1748696265 0 0) 0x348c280 con 0x208a840 -3109> 2013-04-03 10:48:31.087991 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos -3108> 2013-04-03 10:48:31.088008 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 update_from_paxos version 394140 summary v 394140 -3107> 2013-04-03 10:48:31.088024 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3106> 2013-04-03 10:48:31.602609 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 228 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140511 v188) v2 ==== 289+0+0 (1259974118 0 0) 0x2ef9840 con 0x2118420 -3105> 2013-04-03 10:48:31.602692 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140511 v188) v2 from mds.? 10.252.58.31:6800/10908 -3104> 2013-04-03 10:48:31.602706 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140511 v188) v2 -- ?+0 0x34ad600 con 0x2118420 -3103> 2013-04-03 10:48:32.076466 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394140 encode_pending v394141 -3102> 2013-04-03 10:48:32.076498 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427577..427591) queue_proposal bl 725 bytes; ctx = 0x330b110 -3101> 2013-04-03 10:48:32.076505 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427577..427591) propose_queued 427592 725 bytes -3100> 2013-04-03 10:48:32.078854 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427591 fc 0 pn 17700 opn 0) v3 -- ?+0 0x34b2c80 -3099> 2013-04-03 10:48:32.236599 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 122 ==== paxos(accept lc 427591 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (2482669391 0 0) 0x34b2c80 con 0x208a840 -3098> 2013-04-03 10:48:32.236952 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427592 fc 0 pn 17700 opn 0) v3 -- ?+0 0x348c280 -3097> 2013-04-03 10:48:32.236972 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427592 fc 427577 pn 0 opn 0) v3 -- ?+0 0x348c000 -3096> 2013-04-03 10:48:32.237016 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos -3095> 2013-04-03 10:48:32.237053 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos version 394141 summary v 394140 -3094> 2013-04-03 10:48:32.237069 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos latest full 393798 -3093> 2013-04-03 10:48:32.237087 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos applying incremental log 394141 2013-04-03 10:48:30.013436 mon.0 10.252.24.25:6789/0 231 : [INF] pgmap v29154: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s -3092> 2013-04-03 10:48:32.237119 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos applying incremental log 394141 2013-04-03 10:48:31.076907 mon.0 10.252.24.25:6789/0 232 : [INF] pgmap v29155: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5723B/s wr, 0op/s -3091> 2013-04-03 10:48:32.237163 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 check_subs -3090> 2013-04-03 10:48:32.237200 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 create_pending v 394142 -3089> 2013-04-03 10:48:32.237223 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 _updated_log for mon.0 10.252.24.25:6789/0 -3088> 2013-04-03 10:48:32.237237 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 232) v1 -- ?+0 0x28a5700 con 0x1bac160 -3087> 2013-04-03 10:48:32.237289 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos -3086> 2013-04-03 10:48:32.237305 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos version 394141 summary v 394141 -3085> 2013-04-03 10:48:32.237325 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3084> 2013-04-03 10:48:32.237347 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 232) v1 ==== 0+0+0 (0 0 0) 0x28a5700 con 0x1bac160 -3083> 2013-04-03 10:48:32.237355 7fcdffcdb700 10 handle_log_ack log(last 232) v1 -3082> 2013-04-03 10:48:32.237356 7fcdffcdb700 10 logged 2013-04-03 10:48:30.013436 mon.0 10.252.24.25:6789/0 231 : [INF] pgmap v29154: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s -3081> 2013-04-03 10:48:32.237363 7fcdffcdb700 10 logged 2013-04-03 10:48:31.076907 mon.0 10.252.24.25:6789/0 232 : [INF] pgmap v29155: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5723B/s wr, 0op/s -3080> 2013-04-03 10:48:32.249538 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 123 ==== paxos(lease_ack lc 427592 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2998213464 0 0) 0x348c000 con 0x208a840 -3079> 2013-04-03 10:48:32.249582 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos -3078> 2013-04-03 10:48:32.249599 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos version 394141 summary v 394141 -3077> 2013-04-03 10:48:32.249615 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3076> 2013-04-03 10:48:32.949253 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 124 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 226 v188) v2) to leader v1 ==== 560+0+0 (932561037 0 0) 0x34ad340 con 0x208a840 -3075> 2013-04-03 10:48:32.949564 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -3074> 2013-04-03 10:48:32.949575 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x34ad080 from 10.252.24.88:6789/0 -3073> 2013-04-03 10:48:32.949634 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 226 v188) v2 from mds.0 10.252.24.25:6802/3144 -3072> 2013-04-03 10:48:32.949646 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 226 v188) v2 tid 28) v2 -- ?+0 0x33e4900 con 0x208a840 -3071> 2013-04-03 10:48:33.389535 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 check_down_pgs -3070> 2013-04-03 10:48:33.389569 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 v29155: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5723B/s wr, 0op/s -3069> 2013-04-03 10:48:33.389600 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -3068> 2013-04-03 10:48:33.389633 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in -3067> 2013-04-03 10:48:33.389672 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 -3066> 2013-04-03 10:48:33.389704 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos -3065> 2013-04-03 10:48:33.389724 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos version 394141 summary v 394141 -3064> 2013-04-03 10:48:33.389734 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 log -3063> 2013-04-03 10:48:33.389752 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3062> 2013-04-03 10:48:33.389771 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth -3061> 2013-04-03 10:48:33.438003 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 24 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 17 v188) v2 ==== 289+0+0 (1374364903 0 0) 0x20e4b80 con 0x3311000 -3060> 2013-04-03 10:48:33.438083 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 17 v188) v2 from mds.? 10.252.24.88:6800/18040 -3059> 2013-04-03 10:48:33.438098 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 17 v188) v2 -- ?+0 0x34ad340 con 0x3311000 -3058> 2013-04-03 10:48:34.651226 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 12 ==== pg_stats(1 pgs tid 74 v 608) v1 ==== 477+0+0 (649215753 0 0) 0x33e4240 con 0x3311160 -3057> 2013-04-03 10:48:34.651304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 preprocess_query pg_stats(1 pgs tid 74 v 608) v1 from osd.1 10.252.24.25:6805/3572 -3056> 2013-04-03 10:48:34.651330 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 prepare_update pg_stats(1 pgs tid 74 v 608) v1 from osd.1 10.252.24.25:6805/3572 -3055> 2013-04-03 10:48:34.651335 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 prepare_pg_stats pg_stats(1 pgs tid 74 v 608) v1 from osd.1 -3054> 2013-04-03 10:48:34.651343 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) -3053> 2013-04-03 10:48:34.701523 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 encode_pending v 29156 -3052> 2013-04-03 10:48:34.701567 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427577..427592) queue_proposal bl 646 bytes; ctx = 0x330b0c0 -3051> 2013-04-03 10:48:34.701576 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427577..427592) propose_queued 427593 646 bytes -3050> 2013-04-03 10:48:34.702391 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427592 fc 0 pn 17700 opn 0) v3 -- ?+0 0x34b2a00 -3049> 2013-04-03 10:48:34.728015 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 125 ==== paxos(accept lc 427592 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (3602702728 0 0) 0x34b2a00 con 0x208a840 -3048> 2013-04-03 10:48:34.728429 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427593 fc 0 pn 17700 opn 0) v3 -- ?+0 0x348c000 -3047> 2013-04-03 10:48:34.728451 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427593 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34b2c80 -3046> 2013-04-03 10:48:34.728541 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29155 update_from_paxos applying incremental 29156 -3045> 2013-04-03 10:48:34.728563 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 v29156: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2576B/s wr, 0op/s -3044> 2013-04-03 10:48:34.728582 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 send_pg_creates to 0 pgs -3043> 2013-04-03 10:48:34.728586 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 update_logger -3042> 2013-04-03 10:48:34.728597 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 create_pending v 29157 -3041> 2013-04-03 10:48:34.728606 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 _updated_stats for osd.1 10.252.24.25:6805/3572 -3040> 2013-04-03 10:48:34.728612 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(1 pgs tid 74) v1 -- ?+0 0x28a5700 con 0x3311160 -3039> 2013-04-03 10:48:34.728630 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 check_osd_map already seen 608 >= 608 -3038> 2013-04-03 10:48:34.728637 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 update_logger -3037> 2013-04-03 10:48:34.728649 7fcdffcdb700 0 log [INF] : pgmap v29156: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2576B/s wr, 0op/s -3036> 2013-04-03 10:48:34.728651 7fcdffcdb700 10 send_log to self -3035> 2013-04-03 10:48:34.728652 7fcdffcdb700 10 log_queue is 1 last_log 233 sent 232 num 1 unsent 1 sending 1 -3034> 2013-04-03 10:48:34.728654 7fcdffcdb700 10 will send 2013-04-03 10:48:34.728650 mon.0 10.252.24.25:6789/0 233 : [INF] pgmap v29156: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2576B/s wr, 0op/s -3033> 2013-04-03 10:48:34.728661 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x1c73c00 -3032> 2013-04-03 10:48:34.728715 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos -3031> 2013-04-03 10:48:34.728736 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos version 394141 summary v 394141 -3030> 2013-04-03 10:48:34.728755 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3029> 2013-04-03 10:48:34.728781 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x1c73c00 con 0x1bac160 -3028> 2013-04-03 10:48:34.728806 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos -3027> 2013-04-03 10:48:34.728822 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos version 394141 summary v 394141 -3026> 2013-04-03 10:48:34.728830 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3025> 2013-04-03 10:48:34.728842 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 preprocess_log log(1 entries) v1 from mon.0 -3024> 2013-04-03 10:48:34.728866 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -3023> 2013-04-03 10:48:34.728879 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 prepare_log log(1 entries) v1 from mon.0 -3022> 2013-04-03 10:48:34.728889 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 logging 2013-04-03 10:48:34.728650 mon.0 10.252.24.25:6789/0 233 : [INF] pgmap v29156: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2576B/s wr, 0op/s -3021> 2013-04-03 10:48:34.751591 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 126 ==== paxos(lease_ack lc 427593 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2289882050 0 0) 0x34b2c80 con 0x208a840 -3020> 2013-04-03 10:48:34.751668 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos -3019> 2013-04-03 10:48:34.751691 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 update_from_paxos version 394141 summary v 394141 -3018> 2013-04-03 10:48:34.751709 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -3017> 2013-04-03 10:48:35.298346 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 127 ==== forward(pg_stats(1 pgs tid 57 v 608) v1) to leader v1 ==== 757+0+0 (2901607031 0 0) 0x2ef9840 con 0x208a840 -3016> 2013-04-03 10:48:35.298378 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -3015> 2013-04-03 10:48:35.298390 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x33e4900 from 10.252.24.88:6789/0 -3014> 2013-04-03 10:48:35.298446 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 preprocess_query pg_stats(1 pgs tid 57 v 608) v1 from osd.2 10.252.24.30:6801/5656 -3013> 2013-04-03 10:48:35.298471 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 prepare_update pg_stats(1 pgs tid 57 v 608) v1 from osd.2 10.252.24.30:6801/5656 -3012> 2013-04-03 10:48:35.298477 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 prepare_pg_stats pg_stats(1 pgs tid 57 v 608) v1 from osd.2 -3011> 2013-04-03 10:48:35.298485 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -3010> 2013-04-03 10:48:35.602737 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 229 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140512 v188) v2 ==== 289+0+0 (4208734938 0 0) 0x2efa8c0 con 0x2118420 -3009> 2013-04-03 10:48:35.602817 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140512 v188) v2 from mds.? 10.252.58.31:6800/10908 -3008> 2013-04-03 10:48:35.602832 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140512 v188) v2 -- ?+0 0x2ef9840 con 0x2118420 -3007> 2013-04-03 10:48:35.728216 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 encode_pending v 29157 -3006> 2013-04-03 10:48:35.728264 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427577..427593) queue_proposal bl 654 bytes; ctx = 0x2c3de70 -3005> 2013-04-03 10:48:35.728273 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427577..427593) propose_queued 427594 654 bytes -3004> 2013-04-03 10:48:35.729657 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427593 fc 0 pn 17700 opn 0) v3 -- ?+0 0x34b2780 -3003> 2013-04-03 10:48:35.729748 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394141 encode_pending v394142 -3002> 2013-04-03 10:48:35.729775 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427577..427593) queue_proposal bl 443 bytes; ctx = 0x2c3d2c0 -3001> 2013-04-03 10:48:35.729781 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos updating c 427577..427593) propose_new_value not active; proposal queued -3000> 2013-04-03 10:48:35.758530 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 128 ==== paxos(accept lc 427593 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (535521961 0 0) 0x34b2780 con 0x208a840 -2999> 2013-04-03 10:48:35.771249 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427594 fc 0 pn 17700 opn 0) v3 -- ?+0 0x34b2c80 -2998> 2013-04-03 10:48:35.771299 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427594 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34b2a00 -2997> 2013-04-03 10:48:35.771782 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29156 update_from_paxos applying incremental 29157 -2996> 2013-04-03 10:48:35.771806 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 v29157: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1750B/s wr, 0op/s -2995> 2013-04-03 10:48:35.771825 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 send_pg_creates to 0 pgs -2994> 2013-04-03 10:48:35.771829 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 update_logger -2993> 2013-04-03 10:48:35.771843 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 create_pending v 29158 -2992> 2013-04-03 10:48:35.771852 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 _updated_stats for osd.2 10.252.24.30:6801/5656 -2991> 2013-04-03 10:48:35.771859 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 57) v1 tid 29) v2 -- ?+0 0x33e4240 con 0x208a840 -2990> 2013-04-03 10:48:35.771878 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 check_osd_map already seen 608 >= 608 -2989> 2013-04-03 10:48:35.771881 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 update_logger -2988> 2013-04-03 10:48:35.771892 7fcdffcdb700 0 log [INF] : pgmap v29157: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1750B/s wr, 0op/s -2987> 2013-04-03 10:48:35.771893 7fcdffcdb700 10 send_log to self -2986> 2013-04-03 10:48:35.771894 7fcdffcdb700 10 log_queue is 2 last_log 234 sent 233 num 2 unsent 1 sending 1 -2985> 2013-04-03 10:48:35.771896 7fcdffcdb700 10 will send 2013-04-03 10:48:34.728650 mon.0 10.252.24.25:6789/0 233 : [INF] pgmap v29156: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2576B/s wr, 0op/s -2984> 2013-04-03 10:48:35.771903 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2874200 -2983> 2013-04-03 10:48:35.771931 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427577..427594) propose_queued 427595 443 bytes -2982> 2013-04-03 10:48:35.772154 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427594 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3476780 -2981> 2013-04-03 10:48:35.772178 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2874200 con 0x1bac160 -2980> 2013-04-03 10:48:35.788102 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 129 ==== paxos(lease_ack lc 427594 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2832928692 0 0) 0x3476780 con 0x208a840 -2979> 2013-04-03 10:48:35.799743 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 130 ==== paxos(accept lc 427594 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (2322180199 0 0) 0x34b2a00 con 0x208a840 -2978> 2013-04-03 10:48:35.800014 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427595 fc 0 pn 17700 opn 0) v3 -- ?+0 0x3476780 -2977> 2013-04-03 10:48:35.800043 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427595 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34b2780 -2976> 2013-04-03 10:48:35.800078 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos -2975> 2013-04-03 10:48:35.800104 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394141 -2974> 2013-04-03 10:48:35.800123 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos latest full 393798 -2973> 2013-04-03 10:48:35.800142 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos applying incremental log 394142 2013-04-03 10:48:34.728650 mon.0 10.252.24.25:6789/0 233 : [INF] pgmap v29156: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2576B/s wr, 0op/s -2972> 2013-04-03 10:48:35.800187 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 check_subs -2971> 2013-04-03 10:48:35.800223 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 create_pending v 394143 -2970> 2013-04-03 10:48:35.800246 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 _updated_log for mon.0 10.252.24.25:6789/0 -2969> 2013-04-03 10:48:35.800258 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 233) v1 -- ?+0 0x28a5a80 con 0x1bac160 -2968> 2013-04-03 10:48:35.800279 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos -2967> 2013-04-03 10:48:35.800294 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 -2966> 2013-04-03 10:48:35.800304 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -2965> 2013-04-03 10:48:35.800315 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 preprocess_log log(1 entries) v1 from mon.0 -2964> 2013-04-03 10:48:35.800335 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 nothing new -2963> 2013-04-03 10:48:35.800386 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos -2962> 2013-04-03 10:48:35.800402 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 -2961> 2013-04-03 10:48:35.800421 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -2960> 2013-04-03 10:48:35.800443 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 233) v1 ==== 0+0+0 (0 0 0) 0x28a5a80 con 0x1bac160 -2959> 2013-04-03 10:48:35.800451 7fcdffcdb700 10 handle_log_ack log(last 233) v1 -2958> 2013-04-03 10:48:35.800453 7fcdffcdb700 10 logged 2013-04-03 10:48:34.728650 mon.0 10.252.24.25:6789/0 233 : [INF] pgmap v29156: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2576B/s wr, 0op/s -2957> 2013-04-03 10:48:35.811752 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 131 ==== paxos(lease_ack lc 427595 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2680875674 0 0) 0x34b2780 con 0x208a840 -2956> 2013-04-03 10:48:35.811841 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos -2955> 2013-04-03 10:48:35.811925 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 -2954> 2013-04-03 10:48:35.811945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -2953> 2013-04-03 10:48:36.949366 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 132 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 227 v188) v2) to leader v1 ==== 560+0+0 (3940599256 0 0) 0x34acdc0 con 0x208a840 -2952> 2013-04-03 10:48:36.949388 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -2951> 2013-04-03 10:48:36.949399 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x34acb00 from 10.252.24.88:6789/0 -2950> 2013-04-03 10:48:36.949456 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 227 v188) v2 from mds.0 10.252.24.25:6802/3144 -2949> 2013-04-03 10:48:36.949468 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 227 v188) v2 tid 30) v2 -- ?+0 0x33e4900 con 0x208a840 -2948> 2013-04-03 10:48:37.438128 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 25 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 18 v188) v2 ==== 289+0+0 (878241120 0 0) 0x20e4600 con 0x3311000 -2947> 2013-04-03 10:48:37.438210 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 18 v188) v2 from mds.? 10.252.24.88:6800/18040 -2946> 2013-04-03 10:48:37.438225 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 18 v188) v2 -- ?+0 0x34acdc0 con 0x3311000 -2945> 2013-04-03 10:48:38.389923 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 check_down_pgs -2944> 2013-04-03 10:48:38.389958 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 v29157: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1750B/s wr, 0op/s -2943> 2013-04-03 10:48:38.389988 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -2942> 2013-04-03 10:48:38.390011 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in -2941> 2013-04-03 10:48:38.390049 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 -2940> 2013-04-03 10:48:38.390081 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos -2939> 2013-04-03 10:48:38.390101 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 -2938> 2013-04-03 10:48:38.390111 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 log -2937> 2013-04-03 10:48:38.390129 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -2936> 2013-04-03 10:48:38.390149 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth -2935> 2013-04-03 10:48:38.800112 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427595 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34b2500 -2934> 2013-04-03 10:48:38.800827 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 133 ==== paxos(lease_ack lc 427595 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2393550447 0 0) 0x34b2500 con 0x208a840 -2933> 2013-04-03 10:48:38.800899 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos -2932> 2013-04-03 10:48:38.800920 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 -2931> 2013-04-03 10:48:38.800945 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -2930> 2013-04-03 10:48:39.602877 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 230 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140513 v188) v2 ==== 289+0+0 (3652357543 0 0) 0x2ef92c0 con 0x2118420 -2929> 2013-04-03 10:48:39.602957 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140513 v188) v2 from mds.? 10.252.58.31:6800/10908 -2928> 2013-04-03 10:48:39.602972 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140513 v188) v2 -- ?+0 0x20e4600 con 0x2118420 -2927> 2013-04-03 10:48:40.949379 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 134 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 228 v188) v2) to leader v1 ==== 560+0+0 (2038964108 0 0) 0x2efa8c0 con 0x208a840 -2926> 2013-04-03 10:48:40.949401 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -2925> 2013-04-03 10:48:40.949414 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x34ac840 from 10.252.24.88:6789/0 -2924> 2013-04-03 10:48:40.949473 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 228 v188) v2 from mds.0 10.252.24.25:6802/3144 -2923> 2013-04-03 10:48:40.949485 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 228 v188) v2 tid 31) v2 -- ?+0 0x33e4240 con 0x208a840 -2922> 2013-04-03 10:48:41.438231 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 26 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 19 v188) v2 ==== 289+0+0 (389437981 0 0) 0x346f600 con 0x3311000 -2921> 2013-04-03 10:48:41.438315 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 19 v188) v2 from mds.? 10.252.24.88:6800/18040 -2920> 2013-04-03 10:48:41.438330 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 19 v188) v2 -- ?+0 0x2efa8c0 con 0x3311000 -2919> 2013-04-03 10:48:41.800177 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427595 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34b2280 -2918> 2013-04-03 10:48:41.800940 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 135 ==== paxos(lease_ack lc 427595 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2708535505 0 0) 0x34b2280 con 0x208a840 -2917> 2013-04-03 10:48:41.801021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos -2916> 2013-04-03 10:48:41.801043 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 -2915> 2013-04-03 10:48:41.801061 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -2914> 2013-04-03 10:48:43.390303 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 v29157: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1750B/s wr, 0op/s -2913> 2013-04-03 10:48:43.390351 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -2912> 2013-04-03 10:48:43.390375 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in -2911> 2013-04-03 10:48:43.390418 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 -2910> 2013-04-03 10:48:43.390450 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos -2909> 2013-04-03 10:48:43.390470 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 -2908> 2013-04-03 10:48:43.390480 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 log -2907> 2013-04-03 10:48:43.390497 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -2906> 2013-04-03 10:48:43.390522 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth -2905> 2013-04-03 10:48:43.602963 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 231 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140514 v188) v2 ==== 289+0+0 (3154201632 0 0) 0x1bae8c0 con 0x2118420 -2904> 2013-04-03 10:48:43.603049 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140514 v188) v2 from mds.? 10.252.58.31:6800/10908 -2903> 2013-04-03 10:48:43.603065 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140514 v188) v2 -- ?+0 0x346f600 con 0x2118420 -2902> 2013-04-03 10:48:44.652137 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 13 ==== pg_stats(1 pgs tid 75 v 608) v1 ==== 477+0+0 (2423229407 0 0) 0x34d96c0 con 0x3311160 -2901> 2013-04-03 10:48:44.652215 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 preprocess_query pg_stats(1 pgs tid 75 v 608) v1 from osd.1 10.252.24.25:6805/3572 -2900> 2013-04-03 10:48:44.652242 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 prepare_update pg_stats(1 pgs tid 75 v 608) v1 from osd.1 10.252.24.25:6805/3572 -2899> 2013-04-03 10:48:44.652248 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 prepare_pg_stats pg_stats(1 pgs tid 75 v 608) v1 from osd.1 -2898> 2013-04-03 10:48:44.652254 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) -2897> 2013-04-03 10:48:44.702434 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 encode_pending v 29158 -2896> 2013-04-03 10:48:44.702482 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427577..427595) queue_proposal bl 646 bytes; ctx = 0x2c3d850 -2895> 2013-04-03 10:48:44.702493 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427577..427595) propose_queued 427596 646 bytes -2894> 2013-04-03 10:48:44.703558 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427595 fc 0 pn 17700 opn 0) v3 -- ?+0 0x34b2000 -2893> 2013-04-03 10:48:44.742032 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 136 ==== paxos(accept lc 427595 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (1991927855 0 0) 0x34b2000 con 0x208a840 -2892> 2013-04-03 10:48:44.742313 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427596 fc 0 pn 17700 opn 0) v3 -- ?+0 0x34b2280 -2891> 2013-04-03 10:48:44.742331 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427596 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34b2500 -2890> 2013-04-03 10:48:44.742409 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29157 update_from_paxos applying incremental 29158 -2889> 2013-04-03 10:48:44.742430 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -2888> 2013-04-03 10:48:44.742448 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 send_pg_creates to 0 pgs -2887> 2013-04-03 10:48:44.742453 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 update_logger -2886> 2013-04-03 10:48:44.742464 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 create_pending v 29159 -2885> 2013-04-03 10:48:44.742473 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 _updated_stats for osd.1 10.252.24.25:6805/3572 -2884> 2013-04-03 10:48:44.742479 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(1 pgs tid 75) v1 -- ?+0 0x28a5a80 con 0x3311160 -2883> 2013-04-03 10:48:44.742490 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 check_osd_map already seen 608 >= 608 -2882> 2013-04-03 10:48:44.742493 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 update_logger -2881> 2013-04-03 10:48:44.742504 7fcdffcdb700 0 log [INF] : pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -2880> 2013-04-03 10:48:44.742505 7fcdffcdb700 10 send_log to self -2879> 2013-04-03 10:48:44.742506 7fcdffcdb700 10 log_queue is 2 last_log 235 sent 233 num 2 unsent 2 sending 2 -2878> 2013-04-03 10:48:44.742508 7fcdffcdb700 10 will send 2013-04-03 10:48:35.771893 mon.0 10.252.24.25:6789/0 234 : [INF] pgmap v29157: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1750B/s wr, 0op/s -2877> 2013-04-03 10:48:44.742516 7fcdffcdb700 10 will send 2013-04-03 10:48:44.742505 mon.0 10.252.24.25:6789/0 235 : [INF] pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -2876> 2013-04-03 10:48:44.742520 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2226600 -2875> 2013-04-03 10:48:44.742571 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos -2874> 2013-04-03 10:48:44.742592 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 -2873> 2013-04-03 10:48:44.742614 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -2872> 2013-04-03 10:48:44.742638 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2226600 con 0x1bac160 -2871> 2013-04-03 10:48:44.742660 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos -2870> 2013-04-03 10:48:44.742676 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 -2869> 2013-04-03 10:48:44.742686 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -2868> 2013-04-03 10:48:44.742697 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 preprocess_log log(2 entries) v1 from mon.0 -2867> 2013-04-03 10:48:44.742726 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -2866> 2013-04-03 10:48:44.742738 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 prepare_log log(2 entries) v1 from mon.0 -2865> 2013-04-03 10:48:44.742749 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 logging 2013-04-03 10:48:35.771893 mon.0 10.252.24.25:6789/0 234 : [INF] pgmap v29157: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1750B/s wr, 0op/s -2864> 2013-04-03 10:48:44.742770 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 logging 2013-04-03 10:48:44.742505 mon.0 10.252.24.25:6789/0 235 : [INF] pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -2863> 2013-04-03 10:48:44.755161 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 137 ==== paxos(lease_ack lc 427596 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (3016102765 0 0) 0x34b2500 con 0x208a840 -2862> 2013-04-03 10:48:44.755211 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos -2861> 2013-04-03 10:48:44.755230 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 update_from_paxos version 394142 summary v 394142 -2860> 2013-04-03 10:48:44.755246 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -2859> 2013-04-03 10:48:44.949664 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 138 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 229 v188) v2) to leader v1 ==== 560+0+0 (3369929341 0 0) 0x2ef92c0 con 0x208a840 -2858> 2013-04-03 10:48:44.949686 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -2857> 2013-04-03 10:48:44.949699 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x34ac580 from 10.252.24.88:6789/0 -2856> 2013-04-03 10:48:44.949756 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 229 v188) v2 from mds.0 10.252.24.25:6802/3144 -2855> 2013-04-03 10:48:44.949768 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 229 v188) v2 tid 32) v2 -- ?+0 0x34d96c0 con 0x208a840 -2854> 2013-04-03 10:48:45.438374 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 27 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 20 v188) v2 ==== 289+0+0 (4282345070 0 0) 0x2ce2b80 con 0x3311000 -2853> 2013-04-03 10:48:45.438455 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 20 v188) v2 from mds.? 10.252.24.88:6800/18040 -2852> 2013-04-03 10:48:45.438469 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 20 v188) v2 -- ?+0 0x20e4080 con 0x3311000 -2851> 2013-04-03 10:48:45.742219 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394142 encode_pending v394143 -2850> 2013-04-03 10:48:45.742252 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos active c 427577..427596) queue_proposal bl 725 bytes; ctx = 0x330b1a0 -2849> 2013-04-03 10:48:45.742259 7fce004dc700 5 mon.hotcpc9039@0(leader sync( leader state none )).paxos(paxos preparing update c 427577..427596) propose_queued 427597 725 bytes -2848> 2013-04-03 10:48:45.743183 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427596 fc 0 pn 17700 opn 0) v3 -- ?+0 0x34e7c80 -2847> 2013-04-03 10:48:45.777334 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 139 ==== paxos(accept lc 427596 fc 0 pn 17700 opn 0) v3 ==== 80+0+0 (3388291243 0 0) 0x34e7c80 con 0x208a840 -2846> 2013-04-03 10:48:45.778079 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427597 fc 0 pn 17700 opn 0) v3 -- ?+0 0x34b2500 -2845> 2013-04-03 10:48:45.778097 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427597 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34b2000 -2844> 2013-04-03 10:48:45.778132 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos -2843> 2013-04-03 10:48:45.778159 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos version 394143 summary v 394142 -2842> 2013-04-03 10:48:45.778173 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos latest full 393798 -2841> 2013-04-03 10:48:45.778190 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos applying incremental log 394143 2013-04-03 10:48:35.771893 mon.0 10.252.24.25:6789/0 234 : [INF] pgmap v29157: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1750B/s wr, 0op/s -2840> 2013-04-03 10:48:45.778219 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos applying incremental log 394143 2013-04-03 10:48:44.742505 mon.0 10.252.24.25:6789/0 235 : [INF] pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -2839> 2013-04-03 10:48:45.778262 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 check_subs -2838> 2013-04-03 10:48:45.778298 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 create_pending v 394144 -2837> 2013-04-03 10:48:45.778321 7fcdffcdb700 7 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 _updated_log for mon.0 10.252.24.25:6789/0 -2836> 2013-04-03 10:48:45.778335 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 235) v1 -- ?+0 0x28a58c0 con 0x1bac160 -2835> 2013-04-03 10:48:45.778388 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos -2834> 2013-04-03 10:48:45.778404 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos version 394143 summary v 394143 -2833> 2013-04-03 10:48:45.778423 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -2832> 2013-04-03 10:48:45.778445 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 235) v1 ==== 0+0+0 (0 0 0) 0x28a58c0 con 0x1bac160 -2831> 2013-04-03 10:48:45.778454 7fcdffcdb700 10 handle_log_ack log(last 235) v1 -2830> 2013-04-03 10:48:45.778455 7fcdffcdb700 10 logged 2013-04-03 10:48:35.771893 mon.0 10.252.24.25:6789/0 234 : [INF] pgmap v29157: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1750B/s wr, 0op/s -2829> 2013-04-03 10:48:45.778462 7fcdffcdb700 10 logged 2013-04-03 10:48:44.742505 mon.0 10.252.24.25:6789/0 235 : [INF] pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -2828> 2013-04-03 10:48:45.790355 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 140 ==== paxos(lease_ack lc 427597 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1004059452 0 0) 0x34b2000 con 0x208a840 -2827> 2013-04-03 10:48:45.790405 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos -2826> 2013-04-03 10:48:45.790424 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos version 394143 summary v 394143 -2825> 2013-04-03 10:48:45.790440 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -2824> 2013-04-03 10:48:47.603111 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 232 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140515 v188) v2 ==== 289+0+0 (2674914141 0 0) 0x2145580 con 0x2118420 -2823> 2013-04-03 10:48:47.603195 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140515 v188) v2 from mds.? 10.252.58.31:6800/10908 -2822> 2013-04-03 10:48:47.603210 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140515 v188) v2 -- ?+0 0x2ce2b80 con 0x2118420 -2821> 2013-04-03 10:48:48.390644 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 check_down_pgs -2820> 2013-04-03 10:48:48.390676 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).pg v29158 v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -2819> 2013-04-03 10:48:48.390706 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -2818> 2013-04-03 10:48:48.390730 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 e608: 5 osds: 5 up, 5 in -2817> 2013-04-03 10:48:48.390767 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).osd e608 min_last_epoch_clean 594 -2816> 2013-04-03 10:48:48.390799 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos -2815> 2013-04-03 10:48:48.390819 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos version 394143 summary v 394143 -2814> 2013-04-03 10:48:48.390828 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 log -2813> 2013-04-03 10:48:48.390844 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -2812> 2013-04-03 10:48:48.390864 7fce004dc700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 auth -2811> 2013-04-03 10:48:48.418548 7fce0612d700 2 -- 10.252.24.25:6789/0 >> 10.252.24.30:6789/0 pipe(0x1f91280 sd=20 :0 s=1 pgs=0 cs=0 l=0).connect couldn't read banner, Success -2810> 2013-04-03 10:48:48.418620 7fce0612d700 2 -- 10.252.24.25:6789/0 >> 10.252.24.30:6789/0 pipe(0x1f91280 sd=20 :0 s=1 pgs=0 cs=0 l=0).fault 0: Success -2809> 2013-04-03 10:48:48.418643 7fce0612d700 0 -- 10.252.24.25:6789/0 >> 10.252.24.30:6789/0 pipe(0x1f91280 sd=20 :0 s=1 pgs=0 cs=0 l=0).fault -2808> 2013-04-03 10:48:48.419124 7fce0612d700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 ms_get_authorizer for mon -2807> 2013-04-03 10:48:48.419990 7fcdfe8d5700 2 -- 10.252.24.25:6789/0 >> 10.252.24.30:6789/0 pipe(0x29b4500 sd=24 :6789 s=4 pgs=0 cs=0 l=0).fault 0: Success -2806> 2013-04-03 10:48:48.420469 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 1 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 ==== 95+0+0 (3455676706 0 0) 0x33b5c00 con 0x208a9a0 -2805> 2013-04-03 10:48:48.420486 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 do not have session, making new one -2804> 2013-04-03 10:48:48.420491 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 ms_dispatch new session MonSession: mon.1 10.252.24.30:6789/0 is open for mon.1 10.252.24.30:6789/0 -2803> 2013-04-03 10:48:48.420497 7fcdffcdb700 5 mon.hotcpc9039@0(leader sync( leader state none )) e5 setting monitor caps on this connection -2802> 2013-04-03 10:48:48.420500 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2801> 2013-04-03 10:48:48.420504 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2800> 2013-04-03 10:48:48.420514 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,2 paxos( fc 427577 lc 427597 )) v4 -- ?+0 0x33b5300 con 0x208a9a0 -2799> 2013-04-03 10:48:48.420563 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 2 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 ==== 95+0+0 (3455676706 0 0) 0x3501300 con 0x208a9a0 -2798> 2013-04-03 10:48:48.420571 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2797> 2013-04-03 10:48:48.420575 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2796> 2013-04-03 10:48:48.420581 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,2 paxos( fc 427577 lc 427597 )) v4 -- ?+0 0x33b5c00 con 0x208a9a0 -2795> 2013-04-03 10:48:48.420590 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 3 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 ==== 95+0+0 (3455676706 0 0) 0x3501000 con 0x208a9a0 -2794> 2013-04-03 10:48:48.420596 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2793> 2013-04-03 10:48:48.420599 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2792> 2013-04-03 10:48:48.420606 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,2 paxos( fc 427577 lc 427597 )) v4 -- ?+0 0x3501300 con 0x208a9a0 -2791> 2013-04-03 10:48:48.420643 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 4 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 ==== 95+0+0 (3455676706 0 0) 0x3501c00 con 0x208a9a0 -2790> 2013-04-03 10:48:48.420651 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2789> 2013-04-03 10:48:48.420654 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2788> 2013-04-03 10:48:48.420660 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,2 paxos( fc 427577 lc 427597 )) v4 -- ?+0 0x3501000 con 0x208a9a0 -2787> 2013-04-03 10:48:48.420707 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 5 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 ==== 95+0+0 (3455676706 0 0) 0x3501900 con 0x208a9a0 -2786> 2013-04-03 10:48:48.420714 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2785> 2013-04-03 10:48:48.420719 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2784> 2013-04-03 10:48:48.420725 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,2 paxos( fc 427577 lc 427597 )) v4 -- ?+0 0x3501c00 con 0x208a9a0 -2783> 2013-04-03 10:48:48.420788 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 6 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 ==== 95+0+0 (3181447375 0 0) 0x1fbd900 con 0x208a9a0 -2782> 2013-04-03 10:48:48.420797 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2781> 2013-04-03 10:48:48.420803 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2780> 2013-04-03 10:48:48.420810 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,2 paxos( fc 427577 lc 427597 )) v4 -- ?+0 0x3501900 con 0x208a9a0 -2779> 2013-04-03 10:48:48.420905 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 7 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 ==== 95+0+0 (641289807 0 0) 0x3501600 con 0x208a9a0 -2778> 2013-04-03 10:48:48.420914 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2777> 2013-04-03 10:48:48.420919 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2776> 2013-04-03 10:48:48.420925 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,2 paxos( fc 427577 lc 427597 )) v4 -- ?+0 0x1fbd900 con 0x208a9a0 -2775> 2013-04-03 10:48:48.420977 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 8 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 ==== 95+0+0 (3455676706 0 0) 0x21ac900 con 0x208a9a0 -2774> 2013-04-03 10:48:48.420988 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2773> 2013-04-03 10:48:48.420993 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2772> 2013-04-03 10:48:48.421000 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,2 paxos( fc 427577 lc 427597 )) v4 -- ?+0 0x3501600 con 0x208a9a0 -2771> 2013-04-03 10:48:48.421010 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 9 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 ==== 95+0+0 (3455676706 0 0) 0x21ac600 con 0x208a9a0 -2770> 2013-04-03 10:48:48.421016 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2769> 2013-04-03 10:48:48.421021 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2768> 2013-04-03 10:48:48.421028 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,2 paxos( fc 427577 lc 427597 )) v4 -- ?+0 0x21ac900 con 0x208a9a0 -2767> 2013-04-03 10:48:48.421059 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 10 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 ==== 95+0+0 (1324595516 0 0) 0x21ac300 con 0x208a9a0 -2766> 2013-04-03 10:48:48.421068 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2765> 2013-04-03 10:48:48.421073 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2764> 2013-04-03 10:48:48.421082 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,2 paxos( fc 427577 lc 427597 )) v4 -- ?+0 0x21ac600 con 0x208a9a0 -2763> 2013-04-03 10:48:48.421102 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 11 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 ==== 95+0+0 (1942599089 0 0) 0x21ac000 con 0x208a9a0 -2762> 2013-04-03 10:48:48.421110 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2761> 2013-04-03 10:48:48.421115 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2760> 2013-04-03 10:48:48.421124 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,2 paxos( fc 427577 lc 427597 )) v4 -- ?+0 0x21ac300 con 0x208a9a0 -2759> 2013-04-03 10:48:48.421168 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 12 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 ==== 95+0+0 (399924269 0 0) 0x21acc00 con 0x208a9a0 -2758> 2013-04-03 10:48:48.421176 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2757> 2013-04-03 10:48:48.421181 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2756> 2013-04-03 10:48:48.421188 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,2 paxos( fc 427577 lc 427597 )) v4 -- ?+0 0x21ac000 con 0x208a9a0 -2755> 2013-04-03 10:48:48.421201 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 13 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 ==== 95+0+0 (3455676706 0 0) 0x3516c00 con 0x208a9a0 -2754> 2013-04-03 10:48:48.421209 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2753> 2013-04-03 10:48:48.421214 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2752> 2013-04-03 10:48:48.421220 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,2 paxos( fc 427577 lc 427597 )) v4 -- ?+0 0x21acc00 con 0x208a9a0 -2751> 2013-04-03 10:48:48.421252 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 14 ==== mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882 paxos( fc 0 lc 0 )) v4 ==== 592+0+0 (3431103800 0 0) 0x3516900 con 0x208a9a0 -2750> 2013-04-03 10:48:48.421260 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882 paxos( fc 0 lc 0 )) v4 -2749> 2013-04-03 10:48:48.421265 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_probe_reply mon.1 10.252.24.30:6789/0mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882 paxos( fc 0 lc 0 )) v4 -2748> 2013-04-03 10:48:48.421270 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 monmap is e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -2747> 2013-04-03 10:48:48.421294 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 15 ==== mon_sync( heartbeat ) v1 ==== 174+0+0 (3528633769 0 0) 0x2ce1b00 con 0x208a9a0 -2746> 2013-04-03 10:48:48.421300 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync mon_sync( heartbeat ) v1 -2745> 2013-04-03 10:48:48.421303 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_heartbeat mon_sync( heartbeat ) v1 -2744> 2013-04-03 10:48:48.421316 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_send_heartbeat mon.1 10.252.24.30:6789/0 reply(1) -2743> 2013-04-03 10:48:48.421321 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( heartbeat_reply ) v1 -- ?+0 0x2145580 -2742> 2013-04-03 10:48:48.778174 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427597 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34e7a00 -2741> 2013-04-03 10:48:48.781820 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 141 ==== paxos(lease_ack lc 427597 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (3976604878 0 0) 0x34e7a00 con 0x208a840 -2740> 2013-04-03 10:48:48.781895 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos -2739> 2013-04-03 10:48:48.781916 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).log v394143 update_from_paxos version 394143 summary v 394143 -2738> 2013-04-03 10:48:48.781934 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).auth v2063 update_from_paxos -2737> 2013-04-03 10:48:48.954324 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 142 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 230 v188) v2) to leader v1 ==== 560+0+0 (1261435420 0 0) 0x1bae8c0 con 0x208a840 -2736> 2013-04-03 10:48:48.954344 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -2735> 2013-04-03 10:48:48.954355 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 mesg 0x34ac2c0 from 10.252.24.88:6789/0 -2734> 2013-04-03 10:48:48.954413 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 230 v188) v2 from mds.0 10.252.24.25:6802/3144 -2733> 2013-04-03 10:48:48.954427 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 230 v188) v2 tid 33) v2 -- ?+0 0x33e4900 con 0x208a840 -2732> 2013-04-03 10:48:49.069168 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 16 ==== mon_sync( finish ) v1 ==== 174+0+0 (3813791064 0 0) 0x2145580 con 0x208a9a0 -2731> 2013-04-03 10:48:49.069193 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync mon_sync( finish ) v1 -2730> 2013-04-03 10:48:49.069204 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 handle_sync_finish mon_sync( finish ) v1 -2729> 2013-04-03 10:48:49.069211 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- mon_sync( finish_reply ) v1 -- ?+0 0x1bae8c0 -2728> 2013-04-03 10:48:49.069219 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish entity(mon.1 10.252.24.30:6789/0) -2727> 2013-04-03 10:48:49.069229 7fcdffcdb700 10 mon.hotcpc9039@0(leader sync( leader state none )) e5 sync_finish no longer a sync leader -2726> 2013-04-03 10:48:49.196583 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 17 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 ==== 95+0+0 (3455676706 0 0) 0x21acc00 con 0x208a9a0 -2725> 2013-04-03 10:48:49.196614 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2724> 2013-04-03 10:48:49.196622 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -2723> 2013-04-03 10:48:49.196633 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,2 paxos( fc 427577 lc 427597 )) v4 -- ?+0 0x3516900 con 0x208a9a0 -2722> 2013-04-03 10:48:49.208733 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 18 ==== election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 propose 347) v3 ==== 537+0+0 (3058594520 0 0) 0x20ad480 con 0x208a9a0 -2721> 2013-04-03 10:48:49.208756 7fcdffcdb700 5 mon.hotcpc9039@0(leader).elector(354) handle_propose from mon.1 -2720> 2013-04-03 10:48:49.208760 7fcdffcdb700 5 mon.hotcpc9039@0(leader).elector(354) got propose from old epoch, quorum is 0,2, mon.1 must have just started -2719> 2013-04-03 10:48:49.208762 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 reset -2718> 2013-04-03 10:48:49.208764 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish -2717> 2013-04-03 10:48:49.208771 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 start_election -2716> 2013-04-03 10:48:49.208773 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 cancel_probe_timeout (none scheduled) -2715> 2013-04-03 10:48:49.208779 7fcdffcdb700 0 log [INF] : mon.hotcpc9039 calling new monitor election -2714> 2013-04-03 10:48:49.208781 7fcdffcdb700 10 send_log to self -2713> 2013-04-03 10:48:49.208782 7fcdffcdb700 10 log_queue is 1 last_log 236 sent 235 num 1 unsent 1 sending 1 -2712> 2013-04-03 10:48:49.208783 7fcdffcdb700 10 will send 2013-04-03 10:48:49.208780 mon.0 10.252.24.25:6789/0 236 : [INF] mon.hotcpc9039 calling new monitor election -2711> 2013-04-03 10:48:49.208790 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2226a00 -2710> 2013-04-03 10:48:49.208797 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(354) start -- can i be leader? -2709> 2013-04-03 10:48:49.208834 7fcdffcdb700 1 mon.hotcpc9039@0(electing).elector(354) init, last seen epoch 354 -2708> 2013-04-03 10:48:49.208839 7fcdffcdb700 10 mon.hotcpc9039@0(electing).elector(354) bump_epoch 354 to 355 -2707> 2013-04-03 10:48:49.210071 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 propose 355) v3 -- ?+0 0x34d96c0 -2706> 2013-04-03 10:48:49.210090 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 propose 355) v3 -- ?+0 0x33e4240 -2705> 2013-04-03 10:48:49.210104 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2226a00 con 0x1bac160 -2704> 2013-04-03 10:48:49.210140 7fcdffcdb700 1 mon.hotcpc9039@0(electing).paxos(paxos recovering c 427577..427597) is_readable now=2013-04-03 10:48:49.210141 lease_expire=2013-04-03 10:48:53.778169 has v0 lc 427597 -2703> 2013-04-03 10:48:49.217025 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 19 ==== election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 ack 355) v3 ==== 537+0+0 (1006236729 0 0) 0x20fc480 con 0x208a9a0 -2702> 2013-04-03 10:48:49.217047 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(355) handle_ack from mon.1 -2701> 2013-04-03 10:48:49.217051 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(355) so far i have {0=4294967295,1=4294967295} -2700> 2013-04-03 10:48:49.253659 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 143 ==== election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 propose 355) v3 ==== 537+0+0 (3720962647 0 0) 0x33e4900 con 0x208a840 -2699> 2013-04-03 10:48:49.253684 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(355) handle_propose from mon.2 -2698> 2013-04-03 10:48:49.253785 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 144 ==== election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 ack 355) v3 ==== 537+0+0 (1006236729 0 0) 0x20fcb40 con 0x208a840 -2697> 2013-04-03 10:48:49.253798 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(355) handle_ack from mon.2 -2696> 2013-04-03 10:48:49.253801 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(355) so far i have {0=4294967295,1=4294967295,2=4294967295} -2695> 2013-04-03 10:48:49.253805 7fcdffcdb700 10 mon.hotcpc9039@0(electing).elector(355) bump_epoch 355 to 356 -2694> 2013-04-03 10:48:49.254055 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 victory 356) v3 -- ?+0 0x33e4900 -2693> 2013-04-03 10:48:49.254071 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 victory 356) v3 -- ?+0 0x33e4240 -2692> 2013-04-03 10:48:49.254080 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 win_election, epoch 356 quorum is 0,1,2 features are 4294967295 -2691> 2013-04-03 10:48:49.254089 7fcdffcdb700 0 log [INF] : mon.hotcpc9039@0 won leader election with quorum 0,1,2 -2690> 2013-04-03 10:48:49.254090 7fcdffcdb700 10 send_log to self -2689> 2013-04-03 10:48:49.254092 7fcdffcdb700 10 log_queue is 2 last_log 237 sent 236 num 2 unsent 1 sending 1 -2688> 2013-04-03 10:48:49.254094 7fcdffcdb700 10 will send 2013-04-03 10:48:49.208780 mon.0 10.252.24.25:6789/0 236 : [INF] mon.hotcpc9039 calling new monitor election -2687> 2013-04-03 10:48:49.254100 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2874200 -2686> 2013-04-03 10:48:49.254328 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(collect lc 427597 fc 427577 pn 17800 opn 0) v3 -- ?+0 0x34e7a00 -2685> 2013-04-03 10:48:49.254343 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(collect lc 427597 fc 427577 pn 17800 opn 0) v3 -- ?+0 0x29b4500 -2684> 2013-04-03 10:48:49.254353 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish -2683> 2013-04-03 10:48:49.254357 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 resend_routed_requests -2682> 2013-04-03 10:48:49.254359 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 register_cluster_logger - already registered -2681> 2013-04-03 10:48:49.254361 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_start -2680> 2013-04-03 10:48:49.254362 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round curr 0 -2679> 2013-04-03 10:48:49.254364 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round new 1 -2678> 2013-04-03 10:48:49.254365 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck -2677> 2013-04-03 10:48:49.254366 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck start timecheck epoch 356 round 1 -2676> 2013-04-03 10:48:49.254370 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck send time_check( ping e 356 r 1 ) v1 to mon.1 10.252.24.30:6789/0 -2675> 2013-04-03 10:48:49.254374 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- time_check( ping e 356 r 1 ) v1 -- ?+0 0x20fc480 -2674> 2013-04-03 10:48:49.254381 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck send time_check( ping e 356 r 1 ) v1 to mon.2 10.252.24.88:6789/0 -2673> 2013-04-03 10:48:49.254384 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- time_check( ping e 356 r 1 ) v1 -- ?+0 0x34d96c0 -2672> 2013-04-03 10:48:49.254388 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round setting up next event -2671> 2013-04-03 10:48:49.254392 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2874200 con 0x1bac160 -2670> 2013-04-03 10:48:49.254426 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427577..427597) is_readable now=2013-04-03 10:48:49.254427 lease_expire=0.000000 has v0 lc 427597 -2669> 2013-04-03 10:48:49.266633 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 20 ==== paxos(last lc 427597 fc 427577 pn 17800 opn 0) v3 ==== 80+0+0 (923624639 0 0) 0x34e7a00 con 0x208a9a0 -2668> 2013-04-03 10:48:49.266672 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 21 ==== time_check( pong e 356 r 1 ts 2013-04-03 10:48:49.265712 ) v1 ==== 36+0+0 (1625959463 0 0) 0x20fc480 con 0x208a9a0 -2667> 2013-04-03 10:48:49.266683 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck time_check( pong e 356 r 1 ts 2013-04-03 10:48:49.265712 ) v1 -2666> 2013-04-03 10:48:49.266688 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader time_check( pong e 356 r 1 ts 2013-04-03 10:48:49.265712 ) v1 -2665> 2013-04-03 10:48:49.266695 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader from mon.1 10.252.24.30:6789/0 ts 2013-04-03 10:48:49.265712 delta -0.0009799 skew_bound 0 latency 0.012323 -2664> 2013-04-03 10:48:49.277039 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 145 ==== paxos(last lc 427597 fc 427577 pn 17800 opn 0) v3 ==== 80+0+0 (89583132 0 0) 0x29b4500 con 0x208a840 -2663> 2013-04-03 10:48:49.277113 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427597 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34e7a00 -2662> 2013-04-03 10:48:49.277127 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427597 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34b2000 -2661> 2013-04-03 10:48:49.277171 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29158 create_pending v 29159 -2660> 2013-04-03 10:48:49.277185 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29158 check_osd_map already seen 608 >= 608 -2659> 2013-04-03 10:48:49.277187 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29158 update_logger -2658> 2013-04-03 10:48:49.277223 7fcdffcdb700 0 log [INF] : pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -2657> 2013-04-03 10:48:49.277225 7fcdffcdb700 10 send_log to self -2656> 2013-04-03 10:48:49.277226 7fcdffcdb700 10 log_queue is 3 last_log 238 sent 236 num 3 unsent 2 sending 2 -2655> 2013-04-03 10:48:49.277228 7fcdffcdb700 10 will send 2013-04-03 10:48:49.208780 mon.0 10.252.24.25:6789/0 236 : [INF] mon.hotcpc9039 calling new monitor election -2654> 2013-04-03 10:48:49.277235 7fcdffcdb700 10 will send 2013-04-03 10:48:49.254090 mon.0 10.252.24.25:6789/0 237 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,1,2 -2653> 2013-04-03 10:48:49.277239 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x1c73c00 -2652> 2013-04-03 10:48:49.277281 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 create_pending e189 -2651> 2013-04-03 10:48:49.277302 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -2650> 2013-04-03 10:48:49.277318 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 update_logger -2649> 2013-04-03 10:48:49.277324 7fcdffcdb700 0 log [INF] : mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -2648> 2013-04-03 10:48:49.277325 7fcdffcdb700 10 send_log to self -2647> 2013-04-03 10:48:49.277326 7fcdffcdb700 10 log_queue is 4 last_log 239 sent 237 num 4 unsent 2 sending 2 -2646> 2013-04-03 10:48:49.277327 7fcdffcdb700 10 will send 2013-04-03 10:48:49.254090 mon.0 10.252.24.25:6789/0 237 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,1,2 -2645> 2013-04-03 10:48:49.277333 7fcdffcdb700 10 will send 2013-04-03 10:48:49.277225 mon.0 10.252.24.25:6789/0 238 : [INF] pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -2644> 2013-04-03 10:48:49.277336 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x1c15a00 -2643> 2013-04-03 10:48:49.277367 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 create_pending e 609 -2642> 2013-04-03 10:48:49.277372 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 remove_redundant_pg_temp -2641> 2013-04-03 10:48:49.277373 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 remove_down_pg_temp -2640> 2013-04-03 10:48:49.277394 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 update_logger -2639> 2013-04-03 10:48:49.277399 7fcdffcdb700 0 log [INF] : osdmap e608: 5 osds: 5 up, 5 in -2638> 2013-04-03 10:48:49.277400 7fcdffcdb700 10 send_log to self -2637> 2013-04-03 10:48:49.277401 7fcdffcdb700 10 log_queue is 5 last_log 240 sent 238 num 5 unsent 2 sending 2 -2636> 2013-04-03 10:48:49.277402 7fcdffcdb700 10 will send 2013-04-03 10:48:49.277225 mon.0 10.252.24.25:6789/0 238 : [INF] pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -2635> 2013-04-03 10:48:49.277407 7fcdffcdb700 10 will send 2013-04-03 10:48:49.277325 mon.0 10.252.24.25:6789/0 239 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -2634> 2013-04-03 10:48:49.277410 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x287e000 -2633> 2013-04-03 10:48:49.277415 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos -2632> 2013-04-03 10:48:49.277439 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 -2631> 2013-04-03 10:48:49.277462 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 create_pending v 394144 -2630> 2013-04-03 10:48:49.278185 7fcdffcdb700 10 mon.hotcpc9039@0(leader).monmap v5 create_pending monmap epoch 6 -2629> 2013-04-03 10:48:49.278233 7fcdffcdb700 0 log [INF] : monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -2628> 2013-04-03 10:48:49.278235 7fcdffcdb700 10 send_log to self -2627> 2013-04-03 10:48:49.278236 7fcdffcdb700 10 log_queue is 6 last_log 241 sent 239 num 6 unsent 2 sending 2 -2626> 2013-04-03 10:48:49.278238 7fcdffcdb700 10 will send 2013-04-03 10:48:49.277325 mon.0 10.252.24.25:6789/0 239 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -2625> 2013-04-03 10:48:49.278245 7fcdffcdb700 10 will send 2013-04-03 10:48:49.277400 mon.0 10.252.24.25:6789/0 240 : [INF] osdmap e608: 5 osds: 5 up, 5 in -2624> 2013-04-03 10:48:49.278250 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2204800 -2623> 2013-04-03 10:48:49.278257 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2622> 2013-04-03 10:48:49.278282 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 create_pending v 2064 -2621> 2013-04-03 10:48:49.278301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 AuthMonitor::on_active() -2620> 2013-04-03 10:48:49.278325 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos -2619> 2013-04-03 10:48:49.278338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 -2618> 2013-04-03 10:48:49.278346 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -2617> 2013-04-03 10:48:49.278356 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(1 entries) v1 from mon.0 -2616> 2013-04-03 10:48:49.278379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -2615> 2013-04-03 10:48:49.278390 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_log log(1 entries) v1 from mon.0 -2614> 2013-04-03 10:48:49.278399 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.208780 mon.0 10.252.24.25:6789/0 236 : [INF] mon.hotcpc9039 calling new monitor election -2613> 2013-04-03 10:48:49.278440 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos -2612> 2013-04-03 10:48:49.278453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 -2611> 2013-04-03 10:48:49.278460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -2610> 2013-04-03 10:48:49.278470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(1 entries) v1 from mon.0 -2609> 2013-04-03 10:48:49.278487 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 nothing new -2608> 2013-04-03 10:48:49.278524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos -2607> 2013-04-03 10:48:49.278538 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 -2606> 2013-04-03 10:48:49.278552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2605> 2013-04-03 10:48:49.278573 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 146 ==== time_check( pong e 356 r 1 ts 2013-04-03 10:48:49.276057 ) v1 ==== 36+0+0 (1083961200 0 0) 0x34d96c0 con 0x208a840 -2604> 2013-04-03 10:48:49.278584 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck time_check( pong e 356 r 1 ts 2013-04-03 10:48:49.276057 ) v1 -2603> 2013-04-03 10:48:49.278588 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader time_check( pong e 356 r 1 ts 2013-04-03 10:48:49.276057 ) v1 -2602> 2013-04-03 10:48:49.278595 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader from mon.2 10.252.24.88:6789/0 ts 2013-04-03 10:48:49.276057 delta -0.00253391 skew_bound 0 latency 0.024211 -2601> 2013-04-03 10:48:49.278608 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_timecheck_leader got pongs from everybody (3 total) -2600> 2013-04-03 10:48:49.278610 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish_round curr 1 -2599> 2013-04-03 10:48:49.278612 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_report -2598> 2013-04-03 10:48:49.278617 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_report send report to mon.1 10.252.24.30:6789/0 -2597> 2013-04-03 10:48:49.278620 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- time_check( report e 356 r 2 #skews 3 #latencies 3 ) v1 -- ?+0 0x20fc480 -2596> 2013-04-03 10:48:49.278632 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_report send report to mon.2 10.252.24.88:6789/0 -2595> 2013-04-03 10:48:49.278635 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- time_check( report e 356 r 2 #skews 3 #latencies 3 ) v1 -- ?+0 0x20fcb40 -2594> 2013-04-03 10:48:49.278645 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x1c73c00 con 0x1bac160 -2593> 2013-04-03 10:48:49.278668 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos -2592> 2013-04-03 10:48:49.278882 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 -2591> 2013-04-03 10:48:49.278891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -2590> 2013-04-03 10:48:49.278903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(2 entries) v1 from mon.0 -2589> 2013-04-03 10:48:49.278927 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -2588> 2013-04-03 10:48:49.278937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_log log(2 entries) v1 from mon.0 -2587> 2013-04-03 10:48:49.278946 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.208780 mon.0 10.252.24.25:6789/0 236 : [INF] mon.hotcpc9039 calling new monitor election -2586> 2013-04-03 10:48:49.278965 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.254090 mon.0 10.252.24.25:6789/0 237 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,1,2 -2585> 2013-04-03 10:48:49.278992 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 22 ==== paxos(lease_ack lc 427597 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1776666466 0 0) 0x34e7a00 con 0x208a9a0 -2584> 2013-04-03 10:48:49.279024 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos -2583> 2013-04-03 10:48:49.279037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 -2582> 2013-04-03 10:48:49.279051 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2581> 2013-04-03 10:48:49.279067 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 147 ==== paxos(lease_ack lc 427597 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1954469396 0 0) 0x34b2000 con 0x208a840 -2580> 2013-04-03 10:48:49.279100 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos -2579> 2013-04-03 10:48:49.279114 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 -2578> 2013-04-03 10:48:49.279128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2577> 2013-04-03 10:48:49.279144 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x1c15a00 con 0x1bac160 -2576> 2013-04-03 10:48:49.279162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos -2575> 2013-04-03 10:48:49.279177 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 -2574> 2013-04-03 10:48:49.279184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -2573> 2013-04-03 10:48:49.279193 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(2 entries) v1 from mon.0 -2572> 2013-04-03 10:48:49.279216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -2571> 2013-04-03 10:48:49.279226 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_log log(2 entries) v1 from mon.0 -2570> 2013-04-03 10:48:49.279235 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.254090 mon.0 10.252.24.25:6789/0 237 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,1,2 -2569> 2013-04-03 10:48:49.279253 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.277225 mon.0 10.252.24.25:6789/0 238 : [INF] pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -2568> 2013-04-03 10:48:49.279280 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 23 ==== forward(log(1 entries) v1) to leader v1 ==== 524+0+0 (3875813470 0 0) 0x1bae8c0 con 0x208a9a0 -2567> 2013-04-03 10:48:49.279286 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mon.1 10.252.24.30:6789/0 via mon.1 10.252.24.30:6789/0 -2566> 2013-04-03 10:48:49.279292 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x1c15000 from 10.252.24.30:6789/0 -2565> 2013-04-03 10:48:49.279307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos -2564> 2013-04-03 10:48:49.279320 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 -2563> 2013-04-03 10:48:49.279328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(1 entries) v1 from mon.1 10.252.24.30:6789/0 -2562> 2013-04-03 10:48:49.279337 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(1 entries) v1 from mon.1 -2561> 2013-04-03 10:48:49.279347 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_update log(1 entries) v1 from mon.1 10.252.24.30:6789/0 -2560> 2013-04-03 10:48:49.279357 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_log log(1 entries) v1 from mon.1 -2559> 2013-04-03 10:48:49.279365 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.196342 mon.1 10.252.24.30:6789/0 1 : [INF] mon.hotcpc9882 calling new monitor election -2558> 2013-04-03 10:48:49.279386 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 148 ==== forward(log(1 entries) v1) to leader v1 ==== 525+0+0 (3859647374 0 0) 0x2ce1b00 con 0x208a840 -2557> 2013-04-03 10:48:49.279393 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mon.2 10.252.24.88:6789/0 via mon.2 10.252.24.88:6789/0 -2556> 2013-04-03 10:48:49.279399 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x287ea00 from 10.252.24.88:6789/0 -2555> 2013-04-03 10:48:49.279414 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos -2554> 2013-04-03 10:48:49.279427 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 -2553> 2013-04-03 10:48:49.279435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(1 entries) v1 from mon.2 10.252.24.88:6789/0 -2552> 2013-04-03 10:48:49.279445 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(1 entries) v1 from mon.2 -2551> 2013-04-03 10:48:49.279456 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_update log(1 entries) v1 from mon.2 10.252.24.88:6789/0 -2550> 2013-04-03 10:48:49.279466 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_log log(1 entries) v1 from mon.2 -2549> 2013-04-03 10:48:49.279474 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.208409 mon.2 10.252.24.88:6789/0 2 : [INF] mon.hotcpc12533 calling new monitor election -2548> 2013-04-03 10:48:49.279495 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x287e000 con 0x1bac160 -2547> 2013-04-03 10:48:49.279514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos -2546> 2013-04-03 10:48:49.279527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 -2545> 2013-04-03 10:48:49.279535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -2544> 2013-04-03 10:48:49.279544 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(2 entries) v1 from mon.0 -2543> 2013-04-03 10:48:49.279566 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -2542> 2013-04-03 10:48:49.279576 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_log log(2 entries) v1 from mon.0 -2541> 2013-04-03 10:48:49.279584 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.277225 mon.0 10.252.24.25:6789/0 238 : [INF] pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -2540> 2013-04-03 10:48:49.279602 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.277325 mon.0 10.252.24.25:6789/0 239 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -2539> 2013-04-03 10:48:49.279630 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2204800 con 0x1bac160 -2538> 2013-04-03 10:48:49.279648 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos -2537> 2013-04-03 10:48:49.279662 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 update_from_paxos version 394143 summary v 394143 -2536> 2013-04-03 10:48:49.279670 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -2535> 2013-04-03 10:48:49.279679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 preprocess_log log(2 entries) v1 from mon.0 -2534> 2013-04-03 10:48:49.279702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -2533> 2013-04-03 10:48:49.279711 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 prepare_log log(2 entries) v1 from mon.0 -2532> 2013-04-03 10:48:49.279720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.277325 mon.0 10.252.24.25:6789/0 239 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -2531> 2013-04-03 10:48:49.279737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394143 logging 2013-04-03 10:48:49.277400 mon.0 10.252.24.25:6789/0 240 : [INF] osdmap e608: 5 osds: 5 up, 5 in -2530> 2013-04-03 10:48:49.328593 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394143 encode_pending v394144 -2529> 2013-04-03 10:48:49.328628 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427597) queue_proposal bl 1755 bytes; ctx = 0x330b090 -2528> 2013-04-03 10:48:49.328636 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427597) propose_queued 427598 1755 bytes -2527> 2013-04-03 10:48:49.328924 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427597 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34e7780 -2526> 2013-04-03 10:48:49.328947 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427597 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34e7500 -2525> 2013-04-03 10:48:49.339963 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 149 ==== paxos(accept lc 427597 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (4149187755 0 0) 0x34e7500 con 0x208a840 -2524> 2013-04-03 10:48:49.340282 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427598 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34b2000 -2523> 2013-04-03 10:48:49.340298 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427598 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34e7a00 -2522> 2013-04-03 10:48:49.417575 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 24 ==== paxos(accept lc 427597 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (619686046 0 0) 0x34b2000 con 0x208a9a0 -2521> 2013-04-03 10:48:49.417607 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427598 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34e7500 -2520> 2013-04-03 10:48:49.417618 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427598 fc 427577 pn 0 opn 0) v3 -- ?+0 0x29b4500 -2519> 2013-04-03 10:48:49.417635 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos -2518> 2013-04-03 10:48:49.417683 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394143 -2517> 2013-04-03 10:48:49.417697 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos latest full 393798 -2516> 2013-04-03 10:48:49.417718 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 update_from_paxos applying incremental log 394144 2013-04-03 10:48:49.196342 mon.1 10.252.24.30:6789/0 1 : [INF] mon.hotcpc9882 calling new monitor election -2515> 2013-04-03 10:48:49.417744 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 update_from_paxos applying incremental log 394144 2013-04-03 10:48:49.208409 mon.2 10.252.24.88:6789/0 2 : [INF] mon.hotcpc12533 calling new monitor election -2514> 2013-04-03 10:48:49.417766 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 update_from_paxos applying incremental log 394144 2013-04-03 10:48:49.208780 mon.0 10.252.24.25:6789/0 236 : [INF] mon.hotcpc9039 calling new monitor election -2513> 2013-04-03 10:48:49.417784 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 update_from_paxos applying incremental log 394144 2013-04-03 10:48:49.254090 mon.0 10.252.24.25:6789/0 237 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,1,2 -2512> 2013-04-03 10:48:49.417804 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 update_from_paxos applying incremental log 394144 2013-04-03 10:48:49.277225 mon.0 10.252.24.25:6789/0 238 : [INF] pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -2511> 2013-04-03 10:48:49.417832 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 update_from_paxos applying incremental log 394144 2013-04-03 10:48:49.277325 mon.0 10.252.24.25:6789/0 239 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -2510> 2013-04-03 10:48:49.417852 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 update_from_paxos applying incremental log 394144 2013-04-03 10:48:49.277400 mon.0 10.252.24.25:6789/0 240 : [INF] osdmap e608: 5 osds: 5 up, 5 in -2509> 2013-04-03 10:48:49.417906 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 check_subs -2508> 2013-04-03 10:48:49.417945 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 create_pending v 394145 -2507> 2013-04-03 10:48:49.417968 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 _updated_log for mon.0 10.252.24.25:6789/0 -2506> 2013-04-03 10:48:49.417981 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 236) v1 -- ?+0 0x28a58c0 con 0x1bac160 -2505> 2013-04-03 10:48:49.417988 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 _updated_log for mon.0 10.252.24.25:6789/0 -2504> 2013-04-03 10:48:49.418001 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 237) v1 -- ?+0 0x28a5e00 con 0x1bac160 -2503> 2013-04-03 10:48:49.418007 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 _updated_log for mon.0 10.252.24.25:6789/0 -2502> 2013-04-03 10:48:49.418019 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 238) v1 -- ?+0 0x28a5540 con 0x1bac160 -2501> 2013-04-03 10:48:49.418024 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 _updated_log for mon.1 10.252.24.30:6789/0 -2500> 2013-04-03 10:48:49.418035 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- route(log(last 1) v1 tid 1) v2 -- ?+0 0x34d96c0 con 0x208a9a0 -2499> 2013-04-03 10:48:49.418044 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 _updated_log for mon.2 10.252.24.88:6789/0 -2498> 2013-04-03 10:48:49.418057 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(log(last 2) v1 tid 34) v2 -- ?+0 0x33e4900 con 0x208a840 -2497> 2013-04-03 10:48:49.418064 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 _updated_log for mon.0 10.252.24.25:6789/0 -2496> 2013-04-03 10:48:49.418076 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 239) v1 -- ?+0 0x33068c0 con 0x1bac160 -2495> 2013-04-03 10:48:49.418080 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394144 _updated_log for mon.0 10.252.24.25:6789/0 -2494> 2013-04-03 10:48:49.418457 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 240) v1 -- ?+0 0x3306a80 con 0x1bac160 -2493> 2013-04-03 10:48:49.418515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos -2492> 2013-04-03 10:48:49.418531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394144 -2491> 2013-04-03 10:48:49.418548 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2490> 2013-04-03 10:48:49.418570 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 236) v1 ==== 0+0+0 (0 0 0) 0x28a58c0 con 0x1bac160 -2489> 2013-04-03 10:48:49.418577 7fcdffcdb700 10 handle_log_ack log(last 236) v1 -2488> 2013-04-03 10:48:49.418579 7fcdffcdb700 10 logged 2013-04-03 10:48:49.208780 mon.0 10.252.24.25:6789/0 236 : [INF] mon.hotcpc9039 calling new monitor election -2487> 2013-04-03 10:48:49.418586 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 150 ==== paxos(lease_ack lc 427598 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (3960992291 0 0) 0x29b4500 con 0x208a840 -2486> 2013-04-03 10:48:49.418616 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos -2485> 2013-04-03 10:48:49.418632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394144 -2484> 2013-04-03 10:48:49.418647 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2483> 2013-04-03 10:48:49.418663 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 237) v1 ==== 0+0+0 (0 0 0) 0x28a5e00 con 0x1bac160 -2482> 2013-04-03 10:48:49.418669 7fcdffcdb700 10 handle_log_ack log(last 237) v1 -2481> 2013-04-03 10:48:49.418670 7fcdffcdb700 10 logged 2013-04-03 10:48:49.254090 mon.0 10.252.24.25:6789/0 237 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,1,2 -2480> 2013-04-03 10:48:49.418676 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 238) v1 ==== 0+0+0 (0 0 0) 0x28a5540 con 0x1bac160 -2479> 2013-04-03 10:48:49.418682 7fcdffcdb700 10 handle_log_ack log(last 238) v1 -2478> 2013-04-03 10:48:49.418683 7fcdffcdb700 10 logged 2013-04-03 10:48:49.277225 mon.0 10.252.24.25:6789/0 238 : [INF] pgmap v29158: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 819B/s wr, 0op/s -2477> 2013-04-03 10:48:49.418690 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 239) v1 ==== 0+0+0 (0 0 0) 0x33068c0 con 0x1bac160 -2476> 2013-04-03 10:48:49.418695 7fcdffcdb700 10 handle_log_ack log(last 239) v1 -2475> 2013-04-03 10:48:49.418697 7fcdffcdb700 10 logged 2013-04-03 10:48:49.277325 mon.0 10.252.24.25:6789/0 239 : [INF] mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -2474> 2013-04-03 10:48:49.418702 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 240) v1 ==== 0+0+0 (0 0 0) 0x3306a80 con 0x1bac160 -2473> 2013-04-03 10:48:49.418707 7fcdffcdb700 10 handle_log_ack log(last 240) v1 -2472> 2013-04-03 10:48:49.418708 7fcdffcdb700 10 logged 2013-04-03 10:48:49.277400 mon.0 10.252.24.25:6789/0 240 : [INF] osdmap e608: 5 osds: 5 up, 5 in -2471> 2013-04-03 10:48:49.423993 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 25 ==== paxos(lease_ack lc 427598 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (764544613 0 0) 0x34e7500 con 0x208a9a0 -2470> 2013-04-03 10:48:49.424040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos -2469> 2013-04-03 10:48:49.424057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394144 -2468> 2013-04-03 10:48:49.424072 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2467> 2013-04-03 10:48:49.438446 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 28 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 21 v188) v2 ==== 289+0+0 (3696351507 0 0) 0x2ce2600 con 0x3311000 -2466> 2013-04-03 10:48:49.438488 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 21 v188) v2 from mds.? 10.252.24.88:6800/18040 -2465> 2013-04-03 10:48:49.438500 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 21 v188) v2 -- ?+0 0x2ce1b00 con 0x3311000 -2464> 2013-04-03 10:48:50.299668 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 151 ==== forward(pg_stats(1 pgs tid 58 v 608) v1) to leader v1 ==== 757+0+0 (4078216049 0 0) 0x34ac000 con 0x208a840 -2463> 2013-04-03 10:48:50.299695 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -2462> 2013-04-03 10:48:50.299706 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x33e4900 from 10.252.24.88:6789/0 -2461> 2013-04-03 10:48:50.299758 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29158 preprocess_query pg_stats(1 pgs tid 58 v 608) v1 from osd.2 10.252.24.30:6801/5656 -2460> 2013-04-03 10:48:50.299782 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29158 prepare_update pg_stats(1 pgs tid 58 v 608) v1 from osd.2 10.252.24.30:6801/5656 -2459> 2013-04-03 10:48:50.299788 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29158 prepare_pg_stats pg_stats(1 pgs tid 58 v 608) v1 from osd.2 -2458> 2013-04-03 10:48:50.299794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29158 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -2457> 2013-04-03 10:48:50.340178 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29158 encode_pending v 29159 -2456> 2013-04-03 10:48:50.340213 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427598) queue_proposal bl 654 bytes; ctx = 0x2c3d290 -2455> 2013-04-03 10:48:50.340220 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427598) propose_queued 427599 654 bytes -2454> 2013-04-03 10:48:50.342318 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427598 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34e7280 -2453> 2013-04-03 10:48:50.342356 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427598 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34e7000 -2452> 2013-04-03 10:48:50.386408 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 26 ==== paxos(accept lc 427598 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (670427783 0 0) 0x34e7280 con 0x208a9a0 -2451> 2013-04-03 10:48:50.387512 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427599 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34e7500 -2450> 2013-04-03 10:48:50.387528 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427599 fc 0 pn 17800 opn 0) v3 -- ?+0 0x29b4500 -2449> 2013-04-03 10:48:50.387541 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 152 ==== paxos(accept lc 427598 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1699467608 0 0) 0x34e7000 con 0x208a840 -2448> 2013-04-03 10:48:50.387558 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427599 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34e7280 -2447> 2013-04-03 10:48:50.387564 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427599 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34b2000 -2446> 2013-04-03 10:48:50.387619 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29158 update_from_paxos applying incremental 29159 -2445> 2013-04-03 10:48:50.387640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 v29159: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1121B/s wr, 0op/s -2444> 2013-04-03 10:48:50.387659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 send_pg_creates to 0 pgs -2443> 2013-04-03 10:48:50.387662 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 update_logger -2442> 2013-04-03 10:48:50.387672 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 create_pending v 29160 -2441> 2013-04-03 10:48:50.387681 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29159 _updated_stats for osd.2 10.252.24.30:6801/5656 -2440> 2013-04-03 10:48:50.387687 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 58) v1 tid 35) v2 -- ?+0 0x33e4240 con 0x208a840 -2439> 2013-04-03 10:48:50.387698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 check_osd_map already seen 608 >= 608 -2438> 2013-04-03 10:48:50.387700 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 update_logger -2437> 2013-04-03 10:48:50.387710 7fcdffcdb700 0 log [INF] : pgmap v29159: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1121B/s wr, 0op/s -2436> 2013-04-03 10:48:50.387712 7fcdffcdb700 10 send_log to self -2435> 2013-04-03 10:48:50.387713 7fcdffcdb700 10 log_queue is 2 last_log 242 sent 240 num 2 unsent 2 sending 2 -2434> 2013-04-03 10:48:50.387715 7fcdffcdb700 10 will send 2013-04-03 10:48:49.278235 mon.0 10.252.24.25:6789/0 241 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -2433> 2013-04-03 10:48:50.387724 7fcdffcdb700 10 will send 2013-04-03 10:48:50.387711 mon.0 10.252.24.25:6789/0 242 : [INF] pgmap v29159: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1121B/s wr, 0op/s -2432> 2013-04-03 10:48:50.387741 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2204800 -2431> 2013-04-03 10:48:50.387794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos -2430> 2013-04-03 10:48:50.387814 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394144 -2429> 2013-04-03 10:48:50.387831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2428> 2013-04-03 10:48:50.387853 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2204800 con 0x1bac160 -2427> 2013-04-03 10:48:50.387876 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos -2426> 2013-04-03 10:48:50.387891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394144 -2425> 2013-04-03 10:48:50.387899 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -2424> 2013-04-03 10:48:50.387909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 preprocess_log log(2 entries) v1 from mon.0 -2423> 2013-04-03 10:48:50.387937 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -2422> 2013-04-03 10:48:50.387949 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 prepare_log log(2 entries) v1 from mon.0 -2421> 2013-04-03 10:48:50.387959 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 logging 2013-04-03 10:48:49.278235 mon.0 10.252.24.25:6789/0 241 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -2420> 2013-04-03 10:48:50.387979 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 logging 2013-04-03 10:48:50.387711 mon.0 10.252.24.25:6789/0 242 : [INF] pgmap v29159: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1121B/s wr, 0op/s -2419> 2013-04-03 10:48:50.400279 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 153 ==== paxos(lease_ack lc 427599 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (799540868 0 0) 0x34b2000 con 0x208a840 -2418> 2013-04-03 10:48:50.400324 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos -2417> 2013-04-03 10:48:50.400342 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394144 -2416> 2013-04-03 10:48:50.400357 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2415> 2013-04-03 10:48:50.400939 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 27 ==== paxos(lease_ack lc 427599 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (347217271 0 0) 0x34e7280 con 0x208a9a0 -2414> 2013-04-03 10:48:50.400973 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos -2413> 2013-04-03 10:48:50.400992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394144 update_from_paxos version 394144 summary v 394144 -2412> 2013-04-03 10:48:50.401006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2411> 2013-04-03 10:48:51.386639 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394144 encode_pending v394145 -2410> 2013-04-03 10:48:51.386669 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427599) queue_proposal bl 732 bytes; ctx = 0x2c3d2a0 -2409> 2013-04-03 10:48:51.386676 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427599) propose_queued 427600 732 bytes -2408> 2013-04-03 10:48:51.387647 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427599 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3513c80 -2407> 2013-04-03 10:48:51.387686 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427599 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3513a00 -2406> 2013-04-03 10:48:51.415784 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 154 ==== paxos(accept lc 427599 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (718648750 0 0) 0x3513a00 con 0x208a840 -2405> 2013-04-03 10:48:51.416690 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427600 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34e7280 -2404> 2013-04-03 10:48:51.416705 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427600 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34b2000 -2403> 2013-04-03 10:48:51.421490 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 28 ==== paxos(accept lc 427599 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1925128051 0 0) 0x34e7280 con 0x208a9a0 -2402> 2013-04-03 10:48:51.421525 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427600 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3513a00 -2401> 2013-04-03 10:48:51.421535 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427600 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34e7000 -2400> 2013-04-03 10:48:51.421547 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos -2399> 2013-04-03 10:48:51.421598 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394144 -2398> 2013-04-03 10:48:51.421613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos latest full 393798 -2397> 2013-04-03 10:48:51.421638 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394145 update_from_paxos applying incremental log 394145 2013-04-03 10:48:49.278235 mon.0 10.252.24.25:6789/0 241 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -2396> 2013-04-03 10:48:51.421666 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394145 update_from_paxos applying incremental log 394145 2013-04-03 10:48:50.387711 mon.0 10.252.24.25:6789/0 242 : [INF] pgmap v29159: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1121B/s wr, 0op/s -2395> 2013-04-03 10:48:51.421710 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 check_subs -2394> 2013-04-03 10:48:51.421748 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 create_pending v 394146 -2393> 2013-04-03 10:48:51.421770 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394145 _updated_log for mon.0 10.252.24.25:6789/0 -2392> 2013-04-03 10:48:51.421783 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 242) v1 -- ?+0 0x33068c0 con 0x1bac160 -2391> 2013-04-03 10:48:51.421845 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos -2390> 2013-04-03 10:48:51.421862 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 -2389> 2013-04-03 10:48:51.421882 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2388> 2013-04-03 10:48:51.421902 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 242) v1 ==== 0+0+0 (0 0 0) 0x33068c0 con 0x1bac160 -2387> 2013-04-03 10:48:51.421909 7fcdffcdb700 10 handle_log_ack log(last 242) v1 -2386> 2013-04-03 10:48:51.421910 7fcdffcdb700 10 logged 2013-04-03 10:48:49.278235 mon.0 10.252.24.25:6789/0 241 : [INF] monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0} -2385> 2013-04-03 10:48:51.421917 7fcdffcdb700 10 logged 2013-04-03 10:48:50.387711 mon.0 10.252.24.25:6789/0 242 : [INF] pgmap v29159: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1121B/s wr, 0op/s -2384> 2013-04-03 10:48:51.427448 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 155 ==== paxos(lease_ack lc 427600 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (986945175 0 0) 0x34e7000 con 0x208a840 -2383> 2013-04-03 10:48:51.427483 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos -2382> 2013-04-03 10:48:51.427500 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 -2381> 2013-04-03 10:48:51.427515 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2380> 2013-04-03 10:48:51.446435 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 29 ==== paxos(lease_ack lc 427600 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (404667234 0 0) 0x3513a00 con 0x208a9a0 -2379> 2013-04-03 10:48:51.446513 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos -2378> 2013-04-03 10:48:51.446532 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 -2377> 2013-04-03 10:48:51.446549 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2376> 2013-04-03 10:48:51.603271 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 233 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140516 v188) v2 ==== 289+0+0 (2003226414 0 0) 0x2145dc0 con 0x2118420 -2375> 2013-04-03 10:48:51.603355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140516 v188) v2 from mds.? 10.252.58.31:6800/10908 -2374> 2013-04-03 10:48:51.603369 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140516 v188) v2 -- ?+0 0x34ac000 con 0x2118420 -2373> 2013-04-03 10:48:52.949745 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 156 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 231 v188) v2) to leader v1 ==== 560+0+0 (2010871334 0 0) 0x2a52340 con 0x208a840 -2372> 2013-04-03 10:48:52.949767 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -2371> 2013-04-03 10:48:52.949777 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a51580 from 10.252.24.88:6789/0 -2370> 2013-04-03 10:48:52.949843 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 231 v188) v2 from mds.0 10.252.24.25:6802/3144 -2369> 2013-04-03 10:48:52.949856 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 231 v188) v2 tid 36) v2 -- ?+0 0x24f8b40 con 0x208a840 -2368> 2013-04-03 10:48:53.390992 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29159 check_down_pgs -2367> 2013-04-03 10:48:53.391025 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29159 v29159: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1121B/s wr, 0op/s -2366> 2013-04-03 10:48:53.391054 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -2365> 2013-04-03 10:48:53.391082 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -2364> 2013-04-03 10:48:53.391119 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -2363> 2013-04-03 10:48:53.391150 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos -2362> 2013-04-03 10:48:53.391168 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 -2361> 2013-04-03 10:48:53.391177 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394145 log -2360> 2013-04-03 10:48:53.391194 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2359> 2013-04-03 10:48:53.391213 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -2358> 2013-04-03 10:48:53.438615 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 29 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 22 v188) v2 ==== 289+0+0 (3118602388 0 0) 0x273b600 con 0x3311000 -2357> 2013-04-03 10:48:53.438689 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 22 v188) v2 from mds.? 10.252.24.88:6800/18040 -2356> 2013-04-03 10:48:53.438703 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 22 v188) v2 -- ?+0 0x2a52340 con 0x3311000 -2355> 2013-04-03 10:48:53.854644 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 14 ==== pg_stats(1 pgs tid 4 v 608) v1 ==== 485+0+0 (1236959788 0 0) 0x3521480 con 0x33112c0 -2354> 2013-04-03 10:48:53.854726 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 preprocess_query pg_stats(1 pgs tid 4 v 608) v1 from osd.3 10.252.24.88:6804/18376 -2353> 2013-04-03 10:48:53.854751 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 prepare_update pg_stats(1 pgs tid 4 v 608) v1 from osd.3 10.252.24.88:6804/18376 -2352> 2013-04-03 10:48:53.854756 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 prepare_pg_stats pg_stats(1 pgs tid 4 v 608) v1 from osd.3 -2351> 2013-04-03 10:48:53.854763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29159 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -2350> 2013-04-03 10:48:53.904949 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29159 encode_pending v 29160 -2349> 2013-04-03 10:48:53.904995 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427600) queue_proposal bl 654 bytes; ctx = 0x330b010 -2348> 2013-04-03 10:48:53.905003 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427600) propose_queued 427601 654 bytes -2347> 2013-04-03 10:48:53.906571 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427600 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3513780 -2346> 2013-04-03 10:48:53.906593 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427600 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3513500 -2345> 2013-04-03 10:48:53.932402 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 157 ==== paxos(accept lc 427600 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2732556415 0 0) 0x3513500 con 0x208a840 -2344> 2013-04-03 10:48:53.950375 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427601 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3513a00 -2343> 2013-04-03 10:48:53.950394 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427601 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34e7000 -2342> 2013-04-03 10:48:53.971586 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 30 ==== paxos(accept lc 427600 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3772064027 0 0) 0x3513a00 con 0x208a9a0 -2341> 2013-04-03 10:48:53.971604 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427601 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3513500 -2340> 2013-04-03 10:48:53.971613 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427601 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34e7280 -2339> 2013-04-03 10:48:53.971674 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29159 update_from_paxos applying incremental 29160 -2338> 2013-04-03 10:48:53.971695 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 v29160: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1446B/s wr, 0op/s -2337> 2013-04-03 10:48:53.971714 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 send_pg_creates to 0 pgs -2336> 2013-04-03 10:48:53.971718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 update_logger -2335> 2013-04-03 10:48:53.971729 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 create_pending v 29161 -2334> 2013-04-03 10:48:53.971737 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29160 _updated_stats for osd.3 10.252.24.88:6804/18376 -2333> 2013-04-03 10:48:53.971743 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6804/18376 -- pg_stats_ack(1 pgs tid 4) v1 -- ?+0 0x33068c0 con 0x33112c0 -2332> 2013-04-03 10:48:53.971753 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 check_osd_map already seen 608 >= 608 -2331> 2013-04-03 10:48:53.971755 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 update_logger -2330> 2013-04-03 10:48:53.971768 7fcdffcdb700 0 log [INF] : pgmap v29160: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1446B/s wr, 0op/s -2329> 2013-04-03 10:48:53.971770 7fcdffcdb700 10 send_log to self -2328> 2013-04-03 10:48:53.971771 7fcdffcdb700 10 log_queue is 1 last_log 243 sent 242 num 1 unsent 1 sending 1 -2327> 2013-04-03 10:48:53.971772 7fcdffcdb700 10 will send 2013-04-03 10:48:53.971769 mon.0 10.252.24.25:6789/0 243 : [INF] pgmap v29160: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1446B/s wr, 0op/s -2326> 2013-04-03 10:48:53.971780 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x24ebe00 -2325> 2013-04-03 10:48:53.971830 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos -2324> 2013-04-03 10:48:53.971849 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 -2323> 2013-04-03 10:48:53.971867 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2322> 2013-04-03 10:48:53.971888 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x24ebe00 con 0x1bac160 -2321> 2013-04-03 10:48:53.971909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos -2320> 2013-04-03 10:48:53.971925 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 -2319> 2013-04-03 10:48:53.971933 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -2318> 2013-04-03 10:48:53.971943 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 preprocess_log log(1 entries) v1 from mon.0 -2317> 2013-04-03 10:48:53.971966 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -2316> 2013-04-03 10:48:53.971979 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 prepare_log log(1 entries) v1 from mon.0 -2315> 2013-04-03 10:48:53.971989 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 logging 2013-04-03 10:48:53.971769 mon.0 10.252.24.25:6789/0 243 : [INF] pgmap v29160: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1446B/s wr, 0op/s -2314> 2013-04-03 10:48:53.973943 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 158 ==== paxos(lease_ack lc 427601 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (4048561343 0 0) 0x34e7280 con 0x208a840 -2313> 2013-04-03 10:48:53.973992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos -2312> 2013-04-03 10:48:53.974010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 -2311> 2013-04-03 10:48:53.974026 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2310> 2013-04-03 10:48:53.977513 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 31 ==== paxos(lease_ack lc 427601 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1097476049 0 0) 0x3513500 con 0x208a9a0 -2309> 2013-04-03 10:48:53.977557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos -2308> 2013-04-03 10:48:53.978015 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394145 update_from_paxos version 394145 summary v 394145 -2307> 2013-04-03 10:48:53.978040 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2306> 2013-04-03 10:48:54.653224 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 14 ==== pg_stats(1 pgs tid 76 v 608) v1 ==== 477+0+0 (1978873832 0 0) 0x34d9480 con 0x3311160 -2305> 2013-04-03 10:48:54.653298 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 preprocess_query pg_stats(1 pgs tid 76 v 608) v1 from osd.1 10.252.24.25:6805/3572 -2304> 2013-04-03 10:48:54.653324 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 prepare_update pg_stats(1 pgs tid 76 v 608) v1 from osd.1 10.252.24.25:6805/3572 -2303> 2013-04-03 10:48:54.653329 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 prepare_pg_stats pg_stats(1 pgs tid 76 v 608) v1 from osd.1 -2302> 2013-04-03 10:48:54.653336 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29160 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) -2301> 2013-04-03 10:48:54.932622 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394145 encode_pending v394146 -2300> 2013-04-03 10:48:54.932661 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427601) queue_proposal bl 443 bytes; ctx = 0x2c3d270 -2299> 2013-04-03 10:48:54.932675 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427601) propose_queued 427602 443 bytes -2298> 2013-04-03 10:48:54.933809 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427601 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3513280 -2297> 2013-04-03 10:48:54.933850 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427601 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3513000 -2296> 2013-04-03 10:48:54.933923 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29160 encode_pending v 29161 -2295> 2013-04-03 10:48:54.933958 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427577..427601) queue_proposal bl 646 bytes; ctx = 0x330b1f0 -2294> 2013-04-03 10:48:54.933968 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427577..427601) propose_new_value not active; proposal queued -2293> 2013-04-03 10:48:54.954698 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 159 ==== paxos(accept lc 427601 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3672734761 0 0) 0x3513000 con 0x208a840 -2292> 2013-04-03 10:48:54.958290 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427602 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3513500 -2291> 2013-04-03 10:48:54.958326 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427602 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34e7280 -2290> 2013-04-03 10:48:54.965148 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 32 ==== paxos(accept lc 427601 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2219551519 0 0) 0x3513500 con 0x208a9a0 -2289> 2013-04-03 10:48:54.965162 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427602 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3513000 -2288> 2013-04-03 10:48:54.965171 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427602 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3513a00 -2287> 2013-04-03 10:48:54.965184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos -2286> 2013-04-03 10:48:54.965222 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos version 394146 summary v 394145 -2285> 2013-04-03 10:48:54.965237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos latest full 393798 -2284> 2013-04-03 10:48:54.965256 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394146 update_from_paxos applying incremental log 394146 2013-04-03 10:48:53.971769 mon.0 10.252.24.25:6789/0 243 : [INF] pgmap v29160: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1446B/s wr, 0op/s -2283> 2013-04-03 10:48:54.965304 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 check_subs -2282> 2013-04-03 10:48:54.965342 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 create_pending v 394147 -2281> 2013-04-03 10:48:54.965365 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394146 _updated_log for mon.0 10.252.24.25:6789/0 -2280> 2013-04-03 10:48:54.965377 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 243) v1 -- ?+0 0x28a5e00 con 0x1bac160 -2279> 2013-04-03 10:48:54.965402 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427602) propose_queued 427603 646 bytes -2278> 2013-04-03 10:48:54.966562 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427602 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34e7c80 -2277> 2013-04-03 10:48:54.966576 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427602 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34b2780 -2276> 2013-04-03 10:48:54.966590 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 243) v1 ==== 0+0+0 (0 0 0) 0x28a5e00 con 0x1bac160 -2275> 2013-04-03 10:48:54.966597 7fcdffcdb700 10 handle_log_ack log(last 243) v1 -2274> 2013-04-03 10:48:54.966599 7fcdffcdb700 10 logged 2013-04-03 10:48:53.971769 mon.0 10.252.24.25:6789/0 243 : [INF] pgmap v29160: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1446B/s wr, 0op/s -2273> 2013-04-03 10:48:54.967698 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 160 ==== paxos(lease_ack lc 427602 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2399509258 0 0) 0x34b2780 con 0x208a840 -2272> 2013-04-03 10:48:54.979384 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 161 ==== paxos(accept lc 427602 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3471931319 0 0) 0x3513a00 con 0x208a840 -2271> 2013-04-03 10:48:54.979697 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427603 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34b2780 -2270> 2013-04-03 10:48:54.979713 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427603 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3513500 -2269> 2013-04-03 10:48:54.990163 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 33 ==== paxos(lease_ack lc 427602 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (4209913728 0 0) 0x34b2780 con 0x208a9a0 -2268> 2013-04-03 10:48:54.995783 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 34 ==== paxos(accept lc 427602 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3739535037 0 0) 0x34e7c80 con 0x208a9a0 -2267> 2013-04-03 10:48:54.995795 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427603 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34b2780 -2266> 2013-04-03 10:48:54.995804 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427603 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3513a00 -2265> 2013-04-03 10:48:54.995860 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29160 update_from_paxos applying incremental 29161 -2264> 2013-04-03 10:48:54.995882 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 v29161: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1114B/s wr, 0op/s -2263> 2013-04-03 10:48:54.995900 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 send_pg_creates to 0 pgs -2262> 2013-04-03 10:48:54.995903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 update_logger -2261> 2013-04-03 10:48:54.995913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 create_pending v 29162 -2260> 2013-04-03 10:48:54.995922 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29161 _updated_stats for osd.1 10.252.24.25:6805/3572 -2259> 2013-04-03 10:48:54.995928 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(1 pgs tid 76) v1 -- ?+0 0x28a5540 con 0x3311160 -2258> 2013-04-03 10:48:54.995938 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 check_osd_map already seen 608 >= 608 -2257> 2013-04-03 10:48:54.995940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 update_logger -2256> 2013-04-03 10:48:54.995950 7fcdffcdb700 0 log [INF] : pgmap v29161: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1114B/s wr, 0op/s -2255> 2013-04-03 10:48:54.995952 7fcdffcdb700 10 send_log to self -2254> 2013-04-03 10:48:54.995953 7fcdffcdb700 10 log_queue is 1 last_log 244 sent 243 num 1 unsent 1 sending 1 -2253> 2013-04-03 10:48:54.995955 7fcdffcdb700 10 will send 2013-04-03 10:48:54.995951 mon.0 10.252.24.25:6789/0 244 : [INF] pgmap v29161: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1114B/s wr, 0op/s -2252> 2013-04-03 10:48:54.995963 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2226400 -2251> 2013-04-03 10:48:54.996014 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos -2250> 2013-04-03 10:48:54.996033 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos version 394146 summary v 394146 -2249> 2013-04-03 10:48:54.996050 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2248> 2013-04-03 10:48:54.996072 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2226400 con 0x1bac160 -2247> 2013-04-03 10:48:54.996094 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos -2246> 2013-04-03 10:48:54.996109 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos version 394146 summary v 394146 -2245> 2013-04-03 10:48:54.996118 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -2244> 2013-04-03 10:48:54.996129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 preprocess_log log(1 entries) v1 from mon.0 -2243> 2013-04-03 10:48:54.996152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -2242> 2013-04-03 10:48:54.996165 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 prepare_log log(1 entries) v1 from mon.0 -2241> 2013-04-03 10:48:54.996175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 logging 2013-04-03 10:48:54.995951 mon.0 10.252.24.25:6789/0 244 : [INF] pgmap v29161: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1114B/s wr, 0op/s -2240> 2013-04-03 10:48:54.996449 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 162 ==== paxos(lease_ack lc 427603 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (3207972381 0 0) 0x3513a00 con 0x208a840 -2239> 2013-04-03 10:48:54.996486 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos -2238> 2013-04-03 10:48:54.996502 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos version 394146 summary v 394146 -2237> 2013-04-03 10:48:54.996518 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2236> 2013-04-03 10:48:55.004102 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 35 ==== paxos(lease_ack lc 427603 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (868600351 0 0) 0x34b2780 con 0x208a9a0 -2235> 2013-04-03 10:48:55.004136 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos -2234> 2013-04-03 10:48:55.004152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394146 update_from_paxos version 394146 summary v 394146 -2233> 2013-04-03 10:48:55.004168 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2232> 2013-04-03 10:48:55.488808 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 234 ==== mon_subscribe({mdsmap=189+,monmap=6+}) v2 ==== 42+0+0 (3152986420 0 0) 0x2224540 con 0x2118420 -2231> 2013-04-03 10:48:55.488840 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 -2230> 2013-04-03 10:48:55.488852 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -2229> 2013-04-03 10:48:55.488855 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.58.31:6800/10908 -- mon_subscribe_ack(300s) v1 -- ?+0 0x28a5e00 -2228> 2013-04-03 10:48:55.603359 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 235 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140517 v188) v2 ==== 289+0+0 (1409839187 0 0) 0x2145000 con 0x2118420 -2227> 2013-04-03 10:48:55.603442 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140517 v188) v2 from mds.? 10.252.58.31:6800/10908 -2226> 2013-04-03 10:48:55.603455 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140517 v188) v2 -- ?+0 0x273b600 con 0x2118420 -2225> 2013-04-03 10:48:55.979577 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394146 encode_pending v394147 -2224> 2013-04-03 10:48:55.979608 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427603) queue_proposal bl 443 bytes; ctx = 0x330b080 -2223> 2013-04-03 10:48:55.979618 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427603) propose_queued 427604 443 bytes -2222> 2013-04-03 10:48:55.980554 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427603 fc 0 pn 17800 opn 0) v3 -- ?+0 0x353dc80 -2221> 2013-04-03 10:48:55.980576 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427603 fc 0 pn 17800 opn 0) v3 -- ?+0 0x353da00 -2220> 2013-04-03 10:48:56.006487 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 163 ==== paxos(accept lc 427603 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3096933024 0 0) 0x353da00 con 0x208a840 -2219> 2013-04-03 10:48:56.017031 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427604 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34b2780 -2218> 2013-04-03 10:48:56.017054 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427604 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3513a00 -2217> 2013-04-03 10:48:56.025270 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 36 ==== paxos(accept lc 427603 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1122587075 0 0) 0x34b2780 con 0x208a9a0 -2216> 2013-04-03 10:48:56.025307 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427604 fc 427577 pn 0 opn 0) v3 -- ?+0 0x353da00 -2215> 2013-04-03 10:48:56.025319 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427604 fc 427577 pn 0 opn 0) v3 -- ?+0 0x34e7c80 -2214> 2013-04-03 10:48:56.025331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos -2213> 2013-04-03 10:48:56.025382 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394146 -2212> 2013-04-03 10:48:56.025397 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos latest full 393798 -2211> 2013-04-03 10:48:56.025418 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394147 update_from_paxos applying incremental log 394147 2013-04-03 10:48:54.995951 mon.0 10.252.24.25:6789/0 244 : [INF] pgmap v29161: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1114B/s wr, 0op/s -2210> 2013-04-03 10:48:56.025471 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 check_subs -2209> 2013-04-03 10:48:56.025507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 create_pending v 394148 -2208> 2013-04-03 10:48:56.025529 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394147 _updated_log for mon.0 10.252.24.25:6789/0 -2207> 2013-04-03 10:48:56.025542 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 244) v1 -- ?+0 0x2224540 con 0x1bac160 -2206> 2013-04-03 10:48:56.025597 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos -2205> 2013-04-03 10:48:56.025613 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 -2204> 2013-04-03 10:48:56.025632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2203> 2013-04-03 10:48:56.025653 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 244) v1 ==== 0+0+0 (0 0 0) 0x2224540 con 0x1bac160 -2202> 2013-04-03 10:48:56.025661 7fcdffcdb700 10 handle_log_ack log(last 244) v1 -2201> 2013-04-03 10:48:56.025663 7fcdffcdb700 10 logged 2013-04-03 10:48:54.995951 mon.0 10.252.24.25:6789/0 244 : [INF] pgmap v29161: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1114B/s wr, 0op/s -2200> 2013-04-03 10:48:56.039848 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 164 ==== paxos(lease_ack lc 427604 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (3463044600 0 0) 0x34e7c80 con 0x208a840 -2199> 2013-04-03 10:48:56.039913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos -2198> 2013-04-03 10:48:56.039931 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 -2197> 2013-04-03 10:48:56.039947 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2196> 2013-04-03 10:48:56.050164 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 37 ==== paxos(lease_ack lc 427604 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1330956986 0 0) 0x353da00 con 0x208a9a0 -2195> 2013-04-03 10:48:56.050432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos -2194> 2013-04-03 10:48:56.050453 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 -2193> 2013-04-03 10:48:56.050469 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2192> 2013-04-03 10:48:56.949899 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 165 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 232 v188) v2) to leader v1 ==== 560+0+0 (3314022424 0 0) 0x2145dc0 con 0x208a840 -2191> 2013-04-03 10:48:56.949922 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -2190> 2013-04-03 10:48:56.949935 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a51840 from 10.252.24.88:6789/0 -2189> 2013-04-03 10:48:56.949994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 232 v188) v2 from mds.0 10.252.24.25:6802/3144 -2188> 2013-04-03 10:48:56.950005 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 232 v188) v2 tid 37) v2 -- ?+0 0x34d9480 con 0x208a840 -2187> 2013-04-03 10:48:57.438850 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 30 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 23 v188) v2 ==== 289+0+0 (2592909289 0 0) 0x273a2c0 con 0x3311000 -2186> 2013-04-03 10:48:57.438954 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 23 v188) v2 from mds.? 10.252.24.88:6800/18040 -2185> 2013-04-03 10:48:57.438970 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 23 v188) v2 -- ?+0 0x2145dc0 con 0x3311000 -2184> 2013-04-03 10:48:58.391369 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29161 check_down_pgs -2183> 2013-04-03 10:48:58.391397 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29161 v29161: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1114B/s wr, 0op/s -2182> 2013-04-03 10:48:58.391425 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -2181> 2013-04-03 10:48:58.391447 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -2180> 2013-04-03 10:48:58.391481 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -2179> 2013-04-03 10:48:58.391513 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos -2178> 2013-04-03 10:48:58.391532 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 -2177> 2013-04-03 10:48:58.391540 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394147 log -2176> 2013-04-03 10:48:58.391557 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2175> 2013-04-03 10:48:58.391575 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -2174> 2013-04-03 10:48:59.025373 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427604 fc 427577 pn 0 opn 0) v3 -- ?+0 0x353d780 -2173> 2013-04-03 10:48:59.025398 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427604 fc 427577 pn 0 opn 0) v3 -- ?+0 0x353d500 -2172> 2013-04-03 10:48:59.026101 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 38 ==== paxos(lease_ack lc 427604 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (263975169 0 0) 0x353d780 con 0x208a9a0 -2171> 2013-04-03 10:48:59.026176 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos -2170> 2013-04-03 10:48:59.026196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 -2169> 2013-04-03 10:48:59.026214 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2168> 2013-04-03 10:48:59.026233 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 166 ==== paxos(lease_ack lc 427604 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1335033637 0 0) 0x353d500 con 0x208a840 -2167> 2013-04-03 10:48:59.026264 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos -2166> 2013-04-03 10:48:59.026280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 -2165> 2013-04-03 10:48:59.026295 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2164> 2013-04-03 10:48:59.603532 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 236 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140518 v188) v2 ==== 289+0+0 (834384340 0 0) 0x2145840 con 0x2118420 -2163> 2013-04-03 10:48:59.603612 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140518 v188) v2 from mds.? 10.252.58.31:6800/10908 -2162> 2013-04-03 10:48:59.603625 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140518 v188) v2 -- ?+0 0x273a2c0 con 0x2118420 -2161> 2013-04-03 10:49:00.300709 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 167 ==== forward(pg_stats(1 pgs tid 59 v 608) v1) to leader v1 ==== 757+0+0 (4235320499 0 0) 0x2145000 con 0x208a840 -2160> 2013-04-03 10:49:00.300733 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -2159> 2013-04-03 10:49:00.300745 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34d9480 from 10.252.24.88:6789/0 -2158> 2013-04-03 10:49:00.300797 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 preprocess_query pg_stats(1 pgs tid 59 v 608) v1 from osd.2 10.252.24.30:6801/5656 -2157> 2013-04-03 10:49:00.300821 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 prepare_update pg_stats(1 pgs tid 59 v 608) v1 from osd.2 10.252.24.30:6801/5656 -2156> 2013-04-03 10:49:00.300827 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 prepare_pg_stats pg_stats(1 pgs tid 59 v 608) v1 from osd.2 -2155> 2013-04-03 10:49:00.300833 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29161 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -2154> 2013-04-03 10:49:00.351018 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29161 encode_pending v 29162 -2153> 2013-04-03 10:49:00.351053 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427604) queue_proposal bl 654 bytes; ctx = 0x330b0d0 -2152> 2013-04-03 10:49:00.351062 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427604) propose_queued 427605 654 bytes -2151> 2013-04-03 10:49:00.354207 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427604 fc 0 pn 17800 opn 0) v3 -- ?+0 0x353d280 -2150> 2013-04-03 10:49:00.354238 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427604 fc 0 pn 17800 opn 0) v3 -- ?+0 0x353d000 -2149> 2013-04-03 10:49:00.388572 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 39 ==== paxos(accept lc 427604 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (4079130317 0 0) 0x353d280 con 0x208a9a0 -2148> 2013-04-03 10:49:00.388971 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427605 fc 0 pn 17800 opn 0) v3 -- ?+0 0x353d500 -2147> 2013-04-03 10:49:00.388996 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427605 fc 0 pn 17800 opn 0) v3 -- ?+0 0x353d780 -2146> 2013-04-03 10:49:00.443368 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 168 ==== paxos(accept lc 427604 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1060189141 0 0) 0x353d780 con 0x208a840 -2145> 2013-04-03 10:49:00.443408 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427605 fc 427577 pn 0 opn 0) v3 -- ?+0 0x353d280 -2144> 2013-04-03 10:49:00.443419 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427605 fc 427577 pn 0 opn 0) v3 -- ?+0 0x353da00 -2143> 2013-04-03 10:49:00.443488 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29161 update_from_paxos applying incremental 29162 -2142> 2013-04-03 10:49:00.443507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 v29162: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2541B/s wr, 0op/s -2141> 2013-04-03 10:49:00.443526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 send_pg_creates to 0 pgs -2140> 2013-04-03 10:49:00.443528 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 update_logger -2139> 2013-04-03 10:49:00.443539 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 create_pending v 29163 -2138> 2013-04-03 10:49:00.443547 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29162 _updated_stats for osd.2 10.252.24.30:6801/5656 -2137> 2013-04-03 10:49:00.443553 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 59) v1 tid 38) v2 -- ?+0 0x24f8b40 con 0x208a840 -2136> 2013-04-03 10:49:00.443567 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 check_osd_map already seen 608 >= 608 -2135> 2013-04-03 10:49:00.443571 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 update_logger -2134> 2013-04-03 10:49:00.443581 7fcdffcdb700 0 log [INF] : pgmap v29162: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2541B/s wr, 0op/s -2133> 2013-04-03 10:49:00.443583 7fcdffcdb700 10 send_log to self -2132> 2013-04-03 10:49:00.443584 7fcdffcdb700 10 log_queue is 1 last_log 245 sent 244 num 1 unsent 1 sending 1 -2131> 2013-04-03 10:49:00.443586 7fcdffcdb700 10 will send 2013-04-03 10:49:00.443583 mon.0 10.252.24.25:6789/0 245 : [INF] pgmap v29162: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2541B/s wr, 0op/s -2130> 2013-04-03 10:49:00.443592 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2226400 -2129> 2013-04-03 10:49:00.443643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos -2128> 2013-04-03 10:49:00.443663 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 -2127> 2013-04-03 10:49:00.443680 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2126> 2013-04-03 10:49:00.443701 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2226400 con 0x1bac160 -2125> 2013-04-03 10:49:00.443722 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos -2124> 2013-04-03 10:49:00.443737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 -2123> 2013-04-03 10:49:00.443745 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -2122> 2013-04-03 10:49:00.443755 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 preprocess_log log(1 entries) v1 from mon.0 -2121> 2013-04-03 10:49:00.443778 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -2120> 2013-04-03 10:49:00.443790 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 prepare_log log(1 entries) v1 from mon.0 -2119> 2013-04-03 10:49:00.443800 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 logging 2013-04-03 10:49:00.443583 mon.0 10.252.24.25:6789/0 245 : [INF] pgmap v29162: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2541B/s wr, 0op/s -2118> 2013-04-03 10:49:00.444099 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 40 ==== paxos(lease_ack lc 427605 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1118062854 0 0) 0x353d280 con 0x208a9a0 -2117> 2013-04-03 10:49:00.444134 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos -2116> 2013-04-03 10:49:00.444151 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 -2115> 2013-04-03 10:49:00.444167 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2114> 2013-04-03 10:49:00.454927 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 169 ==== paxos(lease_ack lc 427605 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1385723511 0 0) 0x353da00 con 0x208a840 -2113> 2013-04-03 10:49:00.454960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos -2112> 2013-04-03 10:49:00.454976 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394147 update_from_paxos version 394147 summary v 394147 -2111> 2013-04-03 10:49:00.454991 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2110> 2013-04-03 10:49:00.950065 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 170 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 233 v188) v2) to leader v1 ==== 560+0+0 (418577741 0 0) 0x2a848c0 con 0x208a840 -2109> 2013-04-03 10:49:00.950087 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -2108> 2013-04-03 10:49:00.950097 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83000 from 10.252.24.88:6789/0 -2107> 2013-04-03 10:49:00.950152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 233 v188) v2 from mds.0 10.252.24.25:6802/3144 -2106> 2013-04-03 10:49:00.950164 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 233 v188) v2 tid 39) v2 -- ?+0 0x34d9480 con 0x208a840 -2105> 2013-04-03 10:49:01.388765 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394147 encode_pending v394148 -2104> 2013-04-03 10:49:01.388790 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427605) queue_proposal bl 443 bytes; ctx = 0x2c3d2b0 -2103> 2013-04-03 10:49:01.388797 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427605) propose_queued 427606 443 bytes -2102> 2013-04-03 10:49:01.389986 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427605 fc 0 pn 17800 opn 0) v3 -- ?+0 0x354dc80 -2101> 2013-04-03 10:49:01.390019 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427605 fc 0 pn 17800 opn 0) v3 -- ?+0 0x354da00 -2100> 2013-04-03 10:49:01.420535 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 171 ==== paxos(accept lc 427605 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1097228882 0 0) 0x354da00 con 0x208a840 -2099> 2013-04-03 10:49:01.420879 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427606 fc 0 pn 17800 opn 0) v3 -- ?+0 0x353da00 -2098> 2013-04-03 10:49:01.420896 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427606 fc 0 pn 17800 opn 0) v3 -- ?+0 0x353d280 -2097> 2013-04-03 10:49:01.423537 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 41 ==== paxos(accept lc 427605 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3624389110 0 0) 0x353da00 con 0x208a9a0 -2096> 2013-04-03 10:49:01.423568 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427606 fc 427577 pn 0 opn 0) v3 -- ?+0 0x354da00 -2095> 2013-04-03 10:49:01.423579 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427606 fc 427577 pn 0 opn 0) v3 -- ?+0 0x353d780 -2094> 2013-04-03 10:49:01.423592 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos -2093> 2013-04-03 10:49:01.423650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394147 -2092> 2013-04-03 10:49:01.423666 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos latest full 393798 -2091> 2013-04-03 10:49:01.423686 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394148 update_from_paxos applying incremental log 394148 2013-04-03 10:49:00.443583 mon.0 10.252.24.25:6789/0 245 : [INF] pgmap v29162: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2541B/s wr, 0op/s -2090> 2013-04-03 10:49:01.423739 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 check_subs -2089> 2013-04-03 10:49:01.423776 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 create_pending v 394149 -2088> 2013-04-03 10:49:01.423798 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394148 _updated_log for mon.0 10.252.24.25:6789/0 -2087> 2013-04-03 10:49:01.423811 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 245) v1 -- ?+0 0x28a58c0 con 0x1bac160 -2086> 2013-04-03 10:49:01.423866 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos -2085> 2013-04-03 10:49:01.423882 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 -2084> 2013-04-03 10:49:01.423900 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2083> 2013-04-03 10:49:01.423921 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 245) v1 ==== 0+0+0 (0 0 0) 0x28a58c0 con 0x1bac160 -2082> 2013-04-03 10:49:01.423929 7fcdffcdb700 10 handle_log_ack log(last 245) v1 -2081> 2013-04-03 10:49:01.423930 7fcdffcdb700 10 logged 2013-04-03 10:49:00.443583 mon.0 10.252.24.25:6789/0 245 : [INF] pgmap v29162: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2541B/s wr, 0op/s -2080> 2013-04-03 10:49:01.428749 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 42 ==== paxos(lease_ack lc 427606 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1065859511 0 0) 0x354da00 con 0x208a9a0 -2079> 2013-04-03 10:49:01.428824 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos -2078> 2013-04-03 10:49:01.428843 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 -2077> 2013-04-03 10:49:01.428861 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2076> 2013-04-03 10:49:01.438927 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 31 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 24 v188) v2 ==== 289+0+0 (1813780099 0 0) 0x273ab00 con 0x3311000 -2075> 2013-04-03 10:49:01.438983 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 24 v188) v2 from mds.? 10.252.24.88:6800/18040 -2074> 2013-04-03 10:49:01.438995 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 24 v188) v2 -- ?+0 0x2a848c0 con 0x3311000 -2073> 2013-04-03 10:49:01.445289 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 172 ==== paxos(lease_ack lc 427606 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2552038450 0 0) 0x353d780 con 0x208a840 -2072> 2013-04-03 10:49:01.445331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos -2071> 2013-04-03 10:49:01.445348 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 -2070> 2013-04-03 10:49:01.445363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2069> 2013-04-03 10:49:03.391724 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29162 check_down_pgs -2068> 2013-04-03 10:49:03.391750 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29162 v29162: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2541B/s wr, 0op/s -2067> 2013-04-03 10:49:03.392211 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -2066> 2013-04-03 10:49:03.392236 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -2065> 2013-04-03 10:49:03.392273 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -2064> 2013-04-03 10:49:03.392304 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos -2063> 2013-04-03 10:49:03.392323 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 -2062> 2013-04-03 10:49:03.392332 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394148 log -2061> 2013-04-03 10:49:03.392350 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2060> 2013-04-03 10:49:03.392369 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -2059> 2013-04-03 10:49:03.603638 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 237 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140519 v188) v2 ==== 289+0+0 (315952809 0 0) 0x2a83dc0 con 0x2118420 -2058> 2013-04-03 10:49:03.603711 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140519 v188) v2 from mds.? 10.252.58.31:6800/10908 -2057> 2013-04-03 10:49:03.603723 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140519 v188) v2 -- ?+0 0x273ab00 con 0x2118420 -2056> 2013-04-03 10:49:04.423641 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427606 fc 427577 pn 0 opn 0) v3 -- ?+0 0x354d780 -2055> 2013-04-03 10:49:04.423680 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427606 fc 427577 pn 0 opn 0) v3 -- ?+0 0x354d500 -2054> 2013-04-03 10:49:04.424361 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 43 ==== paxos(lease_ack lc 427606 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1349924991 0 0) 0x354d780 con 0x208a9a0 -2053> 2013-04-03 10:49:04.424437 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos -2052> 2013-04-03 10:49:04.424457 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 -2051> 2013-04-03 10:49:04.424475 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2050> 2013-04-03 10:49:04.424494 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 173 ==== paxos(lease_ack lc 427606 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1179335272 0 0) 0x354d500 con 0x208a840 -2049> 2013-04-03 10:49:04.424524 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos -2048> 2013-04-03 10:49:04.424540 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 -2047> 2013-04-03 10:49:04.424556 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2046> 2013-04-03 10:49:04.950224 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 174 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 234 v188) v2) to leader v1 ==== 560+0+0 (1575971075 0 0) 0x2145000 con 0x208a840 -2045> 2013-04-03 10:49:04.950244 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -2044> 2013-04-03 10:49:04.950254 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a84080 from 10.252.24.88:6789/0 -2043> 2013-04-03 10:49:04.950311 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 234 v188) v2 from mds.0 10.252.24.25:6802/3144 -2042> 2013-04-03 10:49:04.950326 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 234 v188) v2 tid 40) v2 -- ?+0 0x24f8b40 con 0x208a840 -2041> 2013-04-03 10:49:05.439099 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 32 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 25 v188) v2 ==== 289+0+0 (1332914686 0 0) 0x273a840 con 0x3311000 -2040> 2013-04-03 10:49:05.439150 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 25 v188) v2 from mds.? 10.252.24.88:6800/18040 -2039> 2013-04-03 10:49:05.439159 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 25 v188) v2 -- ?+0 0x2145000 con 0x3311000 -2038> 2013-04-03 10:49:07.423714 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427606 fc 427577 pn 0 opn 0) v3 -- ?+0 0x354d280 -2037> 2013-04-03 10:49:07.423754 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427606 fc 427577 pn 0 opn 0) v3 -- ?+0 0x354d000 -2036> 2013-04-03 10:49:07.424444 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 44 ==== paxos(lease_ack lc 427606 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2138907333 0 0) 0x354d280 con 0x208a9a0 -2035> 2013-04-03 10:49:07.424519 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos -2034> 2013-04-03 10:49:07.424539 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 -2033> 2013-04-03 10:49:07.424557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2032> 2013-04-03 10:49:07.424576 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 175 ==== paxos(lease_ack lc 427606 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (3076983771 0 0) 0x354d000 con 0x208a840 -2031> 2013-04-03 10:49:07.424607 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos -2030> 2013-04-03 10:49:07.424623 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 -2029> 2013-04-03 10:49:07.424640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2028> 2013-04-03 10:49:07.603747 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 238 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140520 v188) v2 ==== 289+0+0 (3829776323 0 0) 0x2a83b00 con 0x2118420 -2027> 2013-04-03 10:49:07.603818 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140520 v188) v2 from mds.? 10.252.58.31:6800/10908 -2026> 2013-04-03 10:49:07.603831 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140520 v188) v2 -- ?+0 0x273a840 con 0x2118420 -2025> 2013-04-03 10:49:08.392507 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29162 v29162: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 2541B/s wr, 0op/s -2024> 2013-04-03 10:49:08.392551 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -2023> 2013-04-03 10:49:08.392574 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -2022> 2013-04-03 10:49:08.392617 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -2021> 2013-04-03 10:49:08.392648 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos -2020> 2013-04-03 10:49:08.392667 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 -2019> 2013-04-03 10:49:08.392677 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394148 log -2018> 2013-04-03 10:49:08.392694 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -2017> 2013-04-03 10:49:08.392713 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -2016> 2013-04-03 10:49:08.950350 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 176 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 235 v188) v2) to leader v1 ==== 560+0+0 (1420211968 0 0) 0x2a83dc0 con 0x208a840 -2015> 2013-04-03 10:49:08.950371 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -2014> 2013-04-03 10:49:08.950380 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a84b80 from 10.252.24.88:6789/0 -2013> 2013-04-03 10:49:08.950650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 235 v188) v2 from mds.0 10.252.24.25:6802/3144 -2012> 2013-04-03 10:49:08.950668 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 235 v188) v2 tid 41) v2 -- ?+0 0x34d9480 con 0x208a840 -2011> 2013-04-03 10:49:09.439258 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 33 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 26 v188) v2 ==== 289+0+0 (717318265 0 0) 0x2146b80 con 0x3311000 -2010> 2013-04-03 10:49:09.439338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 26 v188) v2 from mds.? 10.252.24.88:6800/18040 -2009> 2013-04-03 10:49:09.439351 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 26 v188) v2 -- ?+0 0x2a83dc0 con 0x3311000 -2008> 2013-04-03 10:49:09.654997 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 15 ==== pg_stats(1 pgs tid 77 v 608) v1 ==== 477+0+0 (3193061409 0 0) 0x34d9240 con 0x3311160 -2007> 2013-04-03 10:49:09.655067 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 preprocess_query pg_stats(1 pgs tid 77 v 608) v1 from osd.1 10.252.24.25:6805/3572 -2006> 2013-04-03 10:49:09.655092 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 prepare_update pg_stats(1 pgs tid 77 v 608) v1 from osd.1 10.252.24.25:6805/3572 -2005> 2013-04-03 10:49:09.655097 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 prepare_pg_stats pg_stats(1 pgs tid 77 v 608) v1 from osd.1 -2004> 2013-04-03 10:49:09.655103 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29162 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) -2003> 2013-04-03 10:49:09.705312 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29162 encode_pending v 29163 -2002> 2013-04-03 10:49:09.705348 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427606) queue_proposal bl 646 bytes; ctx = 0x330b180 -2001> 2013-04-03 10:49:09.705356 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427606) propose_queued 427607 646 bytes -2000> 2013-04-03 10:49:09.707410 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427606 fc 0 pn 17800 opn 0) v3 -- ?+0 0x355ac80 -1999> 2013-04-03 10:49:09.707441 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427606 fc 0 pn 17800 opn 0) v3 -- ?+0 0x355aa00 -1998> 2013-04-03 10:49:09.732997 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 177 ==== paxos(accept lc 427606 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2041119664 0 0) 0x355aa00 con 0x208a840 -1997> 2013-04-03 10:49:09.733353 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427607 fc 0 pn 17800 opn 0) v3 -- ?+0 0x354d000 -1996> 2013-04-03 10:49:09.733377 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427607 fc 0 pn 17800 opn 0) v3 -- ?+0 0x354d280 -1995> 2013-04-03 10:49:09.752524 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 45 ==== paxos(accept lc 427606 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (4044214824 0 0) 0x354d000 con 0x208a9a0 -1994> 2013-04-03 10:49:09.752535 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427607 fc 427577 pn 0 opn 0) v3 -- ?+0 0x355aa00 -1993> 2013-04-03 10:49:09.752543 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427607 fc 427577 pn 0 opn 0) v3 -- ?+0 0x354d500 -1992> 2013-04-03 10:49:09.752591 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29162 update_from_paxos applying incremental 29163 -1991> 2013-04-03 10:49:09.752610 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 v29163: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1109B/s wr, 0op/s -1990> 2013-04-03 10:49:09.752628 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 send_pg_creates to 0 pgs -1989> 2013-04-03 10:49:09.752631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 update_logger -1988> 2013-04-03 10:49:09.752643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 create_pending v 29164 -1987> 2013-04-03 10:49:09.752652 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29163 _updated_stats for osd.1 10.252.24.25:6805/3572 -1986> 2013-04-03 10:49:09.752657 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(1 pgs tid 77) v1 -- ?+0 0x28a58c0 con 0x3311160 -1985> 2013-04-03 10:49:09.752667 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 check_osd_map already seen 608 >= 608 -1984> 2013-04-03 10:49:09.752669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 update_logger -1983> 2013-04-03 10:49:09.752679 7fcdffcdb700 0 log [INF] : pgmap v29163: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1109B/s wr, 0op/s -1982> 2013-04-03 10:49:09.752681 7fcdffcdb700 10 send_log to self -1981> 2013-04-03 10:49:09.752682 7fcdffcdb700 10 log_queue is 1 last_log 246 sent 245 num 1 unsent 1 sending 1 -1980> 2013-04-03 10:49:09.752684 7fcdffcdb700 10 will send 2013-04-03 10:49:09.752680 mon.0 10.252.24.25:6789/0 246 : [INF] pgmap v29163: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1109B/s wr, 0op/s -1979> 2013-04-03 10:49:09.752694 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x25fb400 -1978> 2013-04-03 10:49:09.752744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos -1977> 2013-04-03 10:49:09.752763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 -1976> 2013-04-03 10:49:09.752781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1975> 2013-04-03 10:49:09.752802 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x25fb400 con 0x1bac160 -1974> 2013-04-03 10:49:09.752824 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos -1973> 2013-04-03 10:49:09.752839 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 -1972> 2013-04-03 10:49:09.752847 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1971> 2013-04-03 10:49:09.752858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 preprocess_log log(1 entries) v1 from mon.0 -1970> 2013-04-03 10:49:09.752880 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1969> 2013-04-03 10:49:09.752894 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 prepare_log log(1 entries) v1 from mon.0 -1968> 2013-04-03 10:49:09.752904 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 logging 2013-04-03 10:49:09.752680 mon.0 10.252.24.25:6789/0 246 : [INF] pgmap v29163: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1109B/s wr, 0op/s -1967> 2013-04-03 10:49:09.753342 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 178 ==== paxos(lease_ack lc 427607 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (3881443562 0 0) 0x354d500 con 0x208a840 -1966> 2013-04-03 10:49:09.753377 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos -1965> 2013-04-03 10:49:09.753393 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 -1964> 2013-04-03 10:49:09.753408 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1963> 2013-04-03 10:49:09.777524 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 46 ==== paxos(lease_ack lc 427607 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (545040580 0 0) 0x355aa00 con 0x208a9a0 -1962> 2013-04-03 10:49:09.777557 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos -1961> 2013-04-03 10:49:09.777574 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394148 update_from_paxos version 394148 summary v 394148 -1960> 2013-04-03 10:49:09.777588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1959> 2013-04-03 10:49:10.733211 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394148 encode_pending v394149 -1958> 2013-04-03 10:49:10.733247 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427607) queue_proposal bl 443 bytes; ctx = 0x330b070 -1957> 2013-04-03 10:49:10.733254 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427607) propose_queued 427608 443 bytes -1956> 2013-04-03 10:49:10.734744 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427607 fc 0 pn 17800 opn 0) v3 -- ?+0 0x355a780 -1955> 2013-04-03 10:49:10.734779 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427607 fc 0 pn 17800 opn 0) v3 -- ?+0 0x355a500 -1954> 2013-04-03 10:49:10.761589 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 179 ==== paxos(accept lc 427607 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1667129715 0 0) 0x355a500 con 0x208a840 -1953> 2013-04-03 10:49:10.767169 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427608 fc 0 pn 17800 opn 0) v3 -- ?+0 0x355aa00 -1952> 2013-04-03 10:49:10.767196 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427608 fc 0 pn 17800 opn 0) v3 -- ?+0 0x354d500 -1951> 2013-04-03 10:49:10.771041 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 47 ==== paxos(accept lc 427607 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1619629537 0 0) 0x355aa00 con 0x208a9a0 -1950> 2013-04-03 10:49:10.771053 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427608 fc 427577 pn 0 opn 0) v3 -- ?+0 0x355a500 -1949> 2013-04-03 10:49:10.771061 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427608 fc 427577 pn 0 opn 0) v3 -- ?+0 0x354d000 -1948> 2013-04-03 10:49:10.771071 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos -1947> 2013-04-03 10:49:10.771114 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394148 -1946> 2013-04-03 10:49:10.771128 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos latest full 393798 -1945> 2013-04-03 10:49:10.771148 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394149 update_from_paxos applying incremental log 394149 2013-04-03 10:49:09.752680 mon.0 10.252.24.25:6789/0 246 : [INF] pgmap v29163: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1109B/s wr, 0op/s -1944> 2013-04-03 10:49:10.771198 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 check_subs -1943> 2013-04-03 10:49:10.771233 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 create_pending v 394150 -1942> 2013-04-03 10:49:10.771256 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394149 _updated_log for mon.0 10.252.24.25:6789/0 -1941> 2013-04-03 10:49:10.771268 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 246) v1 -- ?+0 0x3306380 con 0x1bac160 -1940> 2013-04-03 10:49:10.771320 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos -1939> 2013-04-03 10:49:10.771336 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 -1938> 2013-04-03 10:49:10.771353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1937> 2013-04-03 10:49:10.771373 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 246) v1 ==== 0+0+0 (0 0 0) 0x3306380 con 0x1bac160 -1936> 2013-04-03 10:49:10.771381 7fcdffcdb700 10 handle_log_ack log(last 246) v1 -1935> 2013-04-03 10:49:10.771383 7fcdffcdb700 10 logged 2013-04-03 10:49:09.752680 mon.0 10.252.24.25:6789/0 246 : [INF] pgmap v29163: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1109B/s wr, 0op/s -1934> 2013-04-03 10:49:10.775722 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 48 ==== paxos(lease_ack lc 427608 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (3322702803 0 0) 0x355a500 con 0x208a9a0 -1933> 2013-04-03 10:49:10.775765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos -1932> 2013-04-03 10:49:10.775782 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 -1931> 2013-04-03 10:49:10.775799 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1930> 2013-04-03 10:49:10.793417 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 180 ==== paxos(lease_ack lc 427608 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (416263945 0 0) 0x354d000 con 0x208a840 -1929> 2013-04-03 10:49:10.793460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos -1928> 2013-04-03 10:49:10.793477 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 -1927> 2013-04-03 10:49:10.793492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1926> 2013-04-03 10:49:11.603872 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 239 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140521 v188) v2 ==== 289+0+0 (3341484222 0 0) 0x2a832c0 con 0x2118420 -1925> 2013-04-03 10:49:11.603949 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140521 v188) v2 from mds.? 10.252.58.31:6800/10908 -1924> 2013-04-03 10:49:11.603961 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140521 v188) v2 -- ?+0 0x2146b80 con 0x2118420 -1923> 2013-04-03 10:49:12.950421 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 181 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 236 v188) v2) to leader v1 ==== 560+0+0 (1608974128 0 0) 0x2a83b00 con 0x208a840 -1922> 2013-04-03 10:49:12.950443 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -1921> 2013-04-03 10:49:12.950453 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2ce1580 from 10.252.24.88:6789/0 -1920> 2013-04-03 10:49:12.950508 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 236 v188) v2 from mds.0 10.252.24.25:6802/3144 -1919> 2013-04-03 10:49:12.950520 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 236 v188) v2 tid 42) v2 -- ?+0 0x34d9240 con 0x208a840 -1918> 2013-04-03 10:49:13.392836 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29163 check_down_pgs -1917> 2013-04-03 10:49:13.392870 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29163 v29163: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1109B/s wr, 0op/s -1916> 2013-04-03 10:49:13.392897 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -1915> 2013-04-03 10:49:13.392923 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -1914> 2013-04-03 10:49:13.392960 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -1913> 2013-04-03 10:49:13.392990 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos -1912> 2013-04-03 10:49:13.393009 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 -1911> 2013-04-03 10:49:13.393018 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394149 log -1910> 2013-04-03 10:49:13.393036 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1909> 2013-04-03 10:49:13.393054 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -1908> 2013-04-03 10:49:13.439341 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 34 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 27 v188) v2 ==== 289+0+0 (162518788 0 0) 0x1baeb80 con 0x3311000 -1907> 2013-04-03 10:49:13.439395 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 27 v188) v2 from mds.? 10.252.24.88:6800/18040 -1906> 2013-04-03 10:49:13.439405 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 27 v188) v2 -- ?+0 0x2a83b00 con 0x3311000 -1905> 2013-04-03 10:49:13.771129 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427608 fc 427577 pn 0 opn 0) v3 -- ?+0 0x355a280 -1904> 2013-04-03 10:49:13.771168 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427608 fc 427577 pn 0 opn 0) v3 -- ?+0 0x355a000 -1903> 2013-04-03 10:49:13.771827 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 49 ==== paxos(lease_ack lc 427608 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (34889200 0 0) 0x355a280 con 0x208a9a0 -1902> 2013-04-03 10:49:13.771902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos -1901> 2013-04-03 10:49:13.771922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 -1900> 2013-04-03 10:49:13.771940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1899> 2013-04-03 10:49:13.771959 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 182 ==== paxos(lease_ack lc 427608 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (417321116 0 0) 0x355a000 con 0x208a840 -1898> 2013-04-03 10:49:13.771989 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos -1897> 2013-04-03 10:49:13.772004 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 -1896> 2013-04-03 10:49:13.772019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1895> 2013-04-03 10:49:15.302105 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 183 ==== forward(pg_stats(1 pgs tid 60 v 608) v1) to leader v1 ==== 757+0+0 (3524969579 0 0) 0x2a832c0 con 0x208a840 -1894> 2013-04-03 10:49:15.302133 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -1893> 2013-04-03 10:49:15.302143 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34d9240 from 10.252.24.88:6789/0 -1892> 2013-04-03 10:49:15.302196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 preprocess_query pg_stats(1 pgs tid 60 v 608) v1 from osd.2 10.252.24.30:6801/5656 -1891> 2013-04-03 10:49:15.302220 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 prepare_update pg_stats(1 pgs tid 60 v 608) v1 from osd.2 10.252.24.30:6801/5656 -1890> 2013-04-03 10:49:15.302226 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 prepare_pg_stats pg_stats(1 pgs tid 60 v 608) v1 from osd.2 -1889> 2013-04-03 10:49:15.302232 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29163 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -1888> 2013-04-03 10:49:15.352422 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29163 encode_pending v 29164 -1887> 2013-04-03 10:49:15.352452 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427608) queue_proposal bl 654 bytes; ctx = 0x330b050 -1886> 2013-04-03 10:49:15.352461 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427608) propose_queued 427609 654 bytes -1885> 2013-04-03 10:49:15.354151 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427608 fc 0 pn 17800 opn 0) v3 -- ?+0 0x356bc80 -1884> 2013-04-03 10:49:15.354169 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427608 fc 0 pn 17800 opn 0) v3 -- ?+0 0x356ba00 -1883> 2013-04-03 10:49:15.396749 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 184 ==== paxos(accept lc 427608 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (459483813 0 0) 0x356ba00 con 0x208a840 -1882> 2013-04-03 10:49:15.397054 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427609 fc 0 pn 17800 opn 0) v3 -- ?+0 0x355a000 -1881> 2013-04-03 10:49:15.397078 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427609 fc 0 pn 17800 opn 0) v3 -- ?+0 0x355a280 -1880> 2013-04-03 10:49:15.407495 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 50 ==== paxos(accept lc 427608 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (4051777852 0 0) 0x355a000 con 0x208a9a0 -1879> 2013-04-03 10:49:15.407507 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427609 fc 427577 pn 0 opn 0) v3 -- ?+0 0x356ba00 -1878> 2013-04-03 10:49:15.407516 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427609 fc 427577 pn 0 opn 0) v3 -- ?+0 0x354d000 -1877> 2013-04-03 10:49:15.407558 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29163 update_from_paxos applying incremental 29164 -1876> 2013-04-03 10:49:15.407573 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 v29164: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -1875> 2013-04-03 10:49:15.407591 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 send_pg_creates to 0 pgs -1874> 2013-04-03 10:49:15.407593 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 update_logger -1873> 2013-04-03 10:49:15.407603 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 create_pending v 29165 -1872> 2013-04-03 10:49:15.407612 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29164 _updated_stats for osd.2 10.252.24.30:6801/5656 -1871> 2013-04-03 10:49:15.407619 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 60) v1 tid 43) v2 -- ?+0 0x34d9480 con 0x208a840 -1870> 2013-04-03 10:49:15.407629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 check_osd_map already seen 608 >= 608 -1869> 2013-04-03 10:49:15.407631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 update_logger -1868> 2013-04-03 10:49:15.407641 7fcdffcdb700 0 log [INF] : pgmap v29164: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -1867> 2013-04-03 10:49:15.407643 7fcdffcdb700 10 send_log to self -1866> 2013-04-03 10:49:15.407643 7fcdffcdb700 10 log_queue is 1 last_log 247 sent 246 num 1 unsent 1 sending 1 -1865> 2013-04-03 10:49:15.407645 7fcdffcdb700 10 will send 2013-04-03 10:49:15.407642 mon.0 10.252.24.25:6789/0 247 : [INF] pgmap v29164: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -1864> 2013-04-03 10:49:15.407651 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x25fb400 -1863> 2013-04-03 10:49:15.407699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos -1862> 2013-04-03 10:49:15.407718 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 -1861> 2013-04-03 10:49:15.407737 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1860> 2013-04-03 10:49:15.407758 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x25fb400 con 0x1bac160 -1859> 2013-04-03 10:49:15.407779 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos -1858> 2013-04-03 10:49:15.407794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 -1857> 2013-04-03 10:49:15.407803 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1856> 2013-04-03 10:49:15.407813 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 preprocess_log log(1 entries) v1 from mon.0 -1855> 2013-04-03 10:49:15.407835 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1854> 2013-04-03 10:49:15.407845 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 prepare_log log(1 entries) v1 from mon.0 -1853> 2013-04-03 10:49:15.407854 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 logging 2013-04-03 10:49:15.407642 mon.0 10.252.24.25:6789/0 247 : [INF] pgmap v29164: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -1852> 2013-04-03 10:49:15.408478 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 185 ==== paxos(lease_ack lc 427609 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (902646089 0 0) 0x354d000 con 0x208a840 -1851> 2013-04-03 10:49:15.408512 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos -1850> 2013-04-03 10:49:15.408528 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 -1849> 2013-04-03 10:49:15.408543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1848> 2013-04-03 10:49:15.440808 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 51 ==== paxos(lease_ack lc 427609 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1802129668 0 0) 0x356ba00 con 0x208a9a0 -1847> 2013-04-03 10:49:15.440842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos -1846> 2013-04-03 10:49:15.440858 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394149 update_from_paxos version 394149 summary v 394149 -1845> 2013-04-03 10:49:15.440873 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1844> 2013-04-03 10:49:15.604005 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 240 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140522 v188) v2 ==== 289+0+0 (2727919929 0 0) 0x2a84600 con 0x2118420 -1843> 2013-04-03 10:49:15.604086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140522 v188) v2 from mds.? 10.252.58.31:6800/10908 -1842> 2013-04-03 10:49:15.604099 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140522 v188) v2 -- ?+0 0x2a832c0 con 0x2118420 -1841> 2013-04-03 10:49:16.396952 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394149 encode_pending v394150 -1840> 2013-04-03 10:49:16.396980 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427609) queue_proposal bl 443 bytes; ctx = 0x330b1b0 -1839> 2013-04-03 10:49:16.396987 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427609) propose_queued 427610 443 bytes -1838> 2013-04-03 10:49:16.398119 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427609 fc 0 pn 17800 opn 0) v3 -- ?+0 0x356b780 -1837> 2013-04-03 10:49:16.398157 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427609 fc 0 pn 17800 opn 0) v3 -- ?+0 0x356b500 -1836> 2013-04-03 10:49:16.423956 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 186 ==== paxos(accept lc 427609 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1118693582 0 0) 0x356b500 con 0x208a840 -1835> 2013-04-03 10:49:16.424283 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427610 fc 0 pn 17800 opn 0) v3 -- ?+0 0x356ba00 -1834> 2013-04-03 10:49:16.424310 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427610 fc 0 pn 17800 opn 0) v3 -- ?+0 0x354d000 -1833> 2013-04-03 10:49:16.434345 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 52 ==== paxos(accept lc 427609 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3757612535 0 0) 0x356ba00 con 0x208a9a0 -1832> 2013-04-03 10:49:16.434357 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427610 fc 427577 pn 0 opn 0) v3 -- ?+0 0x356b500 -1831> 2013-04-03 10:49:16.434365 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427610 fc 427577 pn 0 opn 0) v3 -- ?+0 0x355a000 -1830> 2013-04-03 10:49:16.434374 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos -1829> 2013-04-03 10:49:16.434401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394149 -1828> 2013-04-03 10:49:16.434418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos latest full 393798 -1827> 2013-04-03 10:49:16.434435 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394150 update_from_paxos applying incremental log 394150 2013-04-03 10:49:15.407642 mon.0 10.252.24.25:6789/0 247 : [INF] pgmap v29164: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -1826> 2013-04-03 10:49:16.434479 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 check_subs -1825> 2013-04-03 10:49:16.434516 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 create_pending v 394151 -1824> 2013-04-03 10:49:16.434538 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394150 _updated_log for mon.0 10.252.24.25:6789/0 -1823> 2013-04-03 10:49:16.434551 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 247) v1 -- ?+0 0x3306700 con 0x1bac160 -1822> 2013-04-03 10:49:16.434603 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos -1821> 2013-04-03 10:49:16.434618 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 -1820> 2013-04-03 10:49:16.434637 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1819> 2013-04-03 10:49:16.434657 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 247) v1 ==== 0+0+0 (0 0 0) 0x3306700 con 0x1bac160 -1818> 2013-04-03 10:49:16.434665 7fcdffcdb700 10 handle_log_ack log(last 247) v1 -1817> 2013-04-03 10:49:16.434668 7fcdffcdb700 10 logged 2013-04-03 10:49:15.407642 mon.0 10.252.24.25:6789/0 247 : [INF] pgmap v29164: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -1816> 2013-04-03 10:49:16.437009 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 187 ==== paxos(lease_ack lc 427610 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2734311986 0 0) 0x355a000 con 0x208a840 -1815> 2013-04-03 10:49:16.437042 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos -1814> 2013-04-03 10:49:16.437058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 -1813> 2013-04-03 10:49:16.437072 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1812> 2013-04-03 10:49:16.447008 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 53 ==== paxos(lease_ack lc 427610 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (3975568219 0 0) 0x356b500 con 0x208a9a0 -1811> 2013-04-03 10:49:16.447049 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos -1810> 2013-04-03 10:49:16.447066 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 -1809> 2013-04-03 10:49:16.447083 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1808> 2013-04-03 10:49:16.950549 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 188 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 237 v188) v2) to leader v1 ==== 560+0+0 (3073288284 0 0) 0x2ce2340 con 0x208a840 -1807> 2013-04-03 10:49:16.950571 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -1806> 2013-04-03 10:49:16.950581 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2ce1b00 from 10.252.24.88:6789/0 -1805> 2013-04-03 10:49:16.950637 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 237 v188) v2 from mds.0 10.252.24.25:6802/3144 -1804> 2013-04-03 10:49:16.950648 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 237 v188) v2 tid 44) v2 -- ?+0 0x34d9240 con 0x208a840 -1803> 2013-04-03 10:49:17.439504 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 35 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 28 v188) v2 ==== 289+0+0 (3785812855 0 0) 0x273a840 con 0x3311000 -1802> 2013-04-03 10:49:17.439555 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 28 v188) v2 from mds.? 10.252.24.88:6800/18040 -1801> 2013-04-03 10:49:17.439565 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 28 v188) v2 -- ?+0 0x2ce2340 con 0x3311000 -1800> 2013-04-03 10:49:18.393197 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29164 check_down_pgs -1799> 2013-04-03 10:49:18.393228 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29164 v29164: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -1798> 2013-04-03 10:49:18.393256 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -1797> 2013-04-03 10:49:18.393280 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -1796> 2013-04-03 10:49:18.393326 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -1795> 2013-04-03 10:49:18.393359 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos -1794> 2013-04-03 10:49:18.393378 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 -1793> 2013-04-03 10:49:18.393387 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394150 log -1792> 2013-04-03 10:49:18.393405 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1791> 2013-04-03 10:49:18.393423 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -1790> 2013-04-03 10:49:19.434427 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427610 fc 427577 pn 0 opn 0) v3 -- ?+0 0x356b280 -1789> 2013-04-03 10:49:19.434464 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427610 fc 427577 pn 0 opn 0) v3 -- ?+0 0x356b000 -1788> 2013-04-03 10:49:19.435057 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 54 ==== paxos(lease_ack lc 427610 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (4067347737 0 0) 0x356b280 con 0x208a9a0 -1787> 2013-04-03 10:49:19.435127 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos -1786> 2013-04-03 10:49:19.435147 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 -1785> 2013-04-03 10:49:19.435165 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1784> 2013-04-03 10:49:19.435195 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 189 ==== paxos(lease_ack lc 427610 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1896390026 0 0) 0x356b000 con 0x208a840 -1783> 2013-04-03 10:49:19.435520 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos -1782> 2013-04-03 10:49:19.435540 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 -1781> 2013-04-03 10:49:19.435555 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1780> 2013-04-03 10:49:19.604085 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 241 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140523 v188) v2 ==== 289+0+0 (2180415044 0 0) 0x273a000 con 0x2118420 -1779> 2013-04-03 10:49:19.604139 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140523 v188) v2 from mds.? 10.252.58.31:6800/10908 -1778> 2013-04-03 10:49:19.604153 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140523 v188) v2 -- ?+0 0x273a840 con 0x2118420 -1777> 2013-04-03 10:49:20.950783 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 190 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 238 v188) v2) to leader v1 ==== 560+0+0 (886855741 0 0) 0x2a84600 con 0x208a840 -1776> 2013-04-03 10:49:20.950805 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -1775> 2013-04-03 10:49:20.950814 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a528c0 from 10.252.24.88:6789/0 -1774> 2013-04-03 10:49:20.950870 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 238 v188) v2 from mds.0 10.252.24.25:6802/3144 -1773> 2013-04-03 10:49:20.950881 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 238 v188) v2 tid 45) v2 -- ?+0 0x34d9480 con 0x208a840 -1772> 2013-04-03 10:49:21.439604 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 36 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 29 v188) v2 ==== 289+0+0 (3267908618 0 0) 0x273ab00 con 0x3311000 -1771> 2013-04-03 10:49:21.439685 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 29 v188) v2 from mds.? 10.252.24.88:6800/18040 -1770> 2013-04-03 10:49:21.439698 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 29 v188) v2 -- ?+0 0x2a84600 con 0x3311000 -1769> 2013-04-03 10:49:22.434508 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427610 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3579c80 -1768> 2013-04-03 10:49:22.434547 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427610 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3579a00 -1767> 2013-04-03 10:49:22.435166 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 55 ==== paxos(lease_ack lc 427610 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2890809081 0 0) 0x3579c80 con 0x208a9a0 -1766> 2013-04-03 10:49:22.435236 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos -1765> 2013-04-03 10:49:22.435256 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 -1764> 2013-04-03 10:49:22.435274 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1763> 2013-04-03 10:49:22.435293 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 191 ==== paxos(lease_ack lc 427610 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (4116998798 0 0) 0x3579a00 con 0x208a840 -1762> 2013-04-03 10:49:22.435322 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos -1761> 2013-04-03 10:49:22.435338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 -1760> 2013-04-03 10:49:22.435355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1759> 2013-04-03 10:49:23.393560 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29164 v29164: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1365B/s wr, 0op/s -1758> 2013-04-03 10:49:23.393605 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -1757> 2013-04-03 10:49:23.393628 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -1756> 2013-04-03 10:49:23.393671 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -1755> 2013-04-03 10:49:23.393702 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos -1754> 2013-04-03 10:49:23.393721 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 -1753> 2013-04-03 10:49:23.393729 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394150 log -1752> 2013-04-03 10:49:23.393745 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1751> 2013-04-03 10:49:23.393762 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -1750> 2013-04-03 10:49:23.604216 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 242 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140524 v188) v2 ==== 289+0+0 (1778336311 0 0) 0x2cc5340 con 0x2118420 -1749> 2013-04-03 10:49:23.604264 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140524 v188) v2 from mds.? 10.252.58.31:6800/10908 -1748> 2013-04-03 10:49:23.604281 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140524 v188) v2 -- ?+0 0x273ab00 con 0x2118420 -1747> 2013-04-03 10:49:23.857790 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 15 ==== pg_stats(1 pgs tid 5 v 608) v1 ==== 485+0+0 (1054095504 0 0) 0x3521240 con 0x33112c0 -1746> 2013-04-03 10:49:23.857867 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 preprocess_query pg_stats(1 pgs tid 5 v 608) v1 from osd.3 10.252.24.88:6804/18376 -1745> 2013-04-03 10:49:23.857892 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 prepare_update pg_stats(1 pgs tid 5 v 608) v1 from osd.3 10.252.24.88:6804/18376 -1744> 2013-04-03 10:49:23.857897 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 prepare_pg_stats pg_stats(1 pgs tid 5 v 608) v1 from osd.3 -1743> 2013-04-03 10:49:23.857903 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29164 got osd.3 osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -1742> 2013-04-03 10:49:23.908095 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29164 encode_pending v 29165 -1741> 2013-04-03 10:49:23.908127 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427610) queue_proposal bl 654 bytes; ctx = 0x330b240 -1740> 2013-04-03 10:49:23.908136 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427610) propose_queued 427611 654 bytes -1739> 2013-04-03 10:49:23.909165 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427610 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579780 -1738> 2013-04-03 10:49:23.909199 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427610 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579500 -1737> 2013-04-03 10:49:23.933665 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 192 ==== paxos(accept lc 427610 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (4074519298 0 0) 0x3579500 con 0x208a840 -1736> 2013-04-03 10:49:23.949006 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427611 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579a00 -1735> 2013-04-03 10:49:23.949034 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427611 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579c80 -1734> 2013-04-03 10:49:23.951999 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 56 ==== paxos(accept lc 427610 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3372170190 0 0) 0x3579a00 con 0x208a9a0 -1733> 2013-04-03 10:49:23.952011 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427611 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3579500 -1732> 2013-04-03 10:49:23.952019 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427611 fc 427577 pn 0 opn 0) v3 -- ?+0 0x356b000 -1731> 2013-04-03 10:49:23.952060 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29164 update_from_paxos applying incremental 29165 -1730> 2013-04-03 10:49:23.952076 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 v29165: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1225B/s wr, 0op/s -1729> 2013-04-03 10:49:23.952094 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 send_pg_creates to 0 pgs -1728> 2013-04-03 10:49:23.952096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 update_logger -1727> 2013-04-03 10:49:23.952106 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 create_pending v 29166 -1726> 2013-04-03 10:49:23.952114 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29165 _updated_stats for osd.3 10.252.24.88:6804/18376 -1725> 2013-04-03 10:49:23.952120 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6804/18376 -- pg_stats_ack(1 pgs tid 5) v1 -- ?+0 0x3306700 con 0x33112c0 -1724> 2013-04-03 10:49:23.952133 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 check_osd_map already seen 608 >= 608 -1723> 2013-04-03 10:49:23.952136 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 update_logger -1722> 2013-04-03 10:49:23.952147 7fcdffcdb700 0 log [INF] : pgmap v29165: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1225B/s wr, 0op/s -1721> 2013-04-03 10:49:23.952149 7fcdffcdb700 10 send_log to self -1720> 2013-04-03 10:49:23.952150 7fcdffcdb700 10 log_queue is 1 last_log 248 sent 247 num 1 unsent 1 sending 1 -1719> 2013-04-03 10:49:23.952151 7fcdffcdb700 10 will send 2013-04-03 10:49:23.952148 mon.0 10.252.24.25:6789/0 248 : [INF] pgmap v29165: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1225B/s wr, 0op/s -1718> 2013-04-03 10:49:23.952158 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x1c74800 -1717> 2013-04-03 10:49:23.952206 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos -1716> 2013-04-03 10:49:23.952225 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 -1715> 2013-04-03 10:49:23.952245 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1714> 2013-04-03 10:49:23.952267 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x1c74800 con 0x1bac160 -1713> 2013-04-03 10:49:23.952289 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos -1712> 2013-04-03 10:49:23.952304 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 -1711> 2013-04-03 10:49:23.952313 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1710> 2013-04-03 10:49:23.952323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 preprocess_log log(1 entries) v1 from mon.0 -1709> 2013-04-03 10:49:23.952629 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1708> 2013-04-03 10:49:23.952645 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 prepare_log log(1 entries) v1 from mon.0 -1707> 2013-04-03 10:49:23.952655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 logging 2013-04-03 10:49:23.952148 mon.0 10.252.24.25:6789/0 248 : [INF] pgmap v29165: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1225B/s wr, 0op/s -1706> 2013-04-03 10:49:23.962124 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 193 ==== paxos(lease_ack lc 427611 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1853947025 0 0) 0x356b000 con 0x208a840 -1705> 2013-04-03 10:49:23.962161 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos -1704> 2013-04-03 10:49:23.962177 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 -1703> 2013-04-03 10:49:23.962192 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1702> 2013-04-03 10:49:23.977008 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 57 ==== paxos(lease_ack lc 427611 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (3087727605 0 0) 0x3579500 con 0x208a9a0 -1701> 2013-04-03 10:49:23.977043 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos -1700> 2013-04-03 10:49:23.977059 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394150 update_from_paxos version 394150 summary v 394150 -1699> 2013-04-03 10:49:23.977074 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1698> 2013-04-03 10:49:24.656678 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 16 ==== pg_stats(1 pgs tid 78 v 608) v1 ==== 477+0+0 (72613859 0 0) 0x34d9000 con 0x3311160 -1697> 2013-04-03 10:49:24.656740 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 preprocess_query pg_stats(1 pgs tid 78 v 608) v1 from osd.1 10.252.24.25:6805/3572 -1696> 2013-04-03 10:49:24.656763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 prepare_update pg_stats(1 pgs tid 78 v 608) v1 from osd.1 10.252.24.25:6805/3572 -1695> 2013-04-03 10:49:24.656768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 prepare_pg_stats pg_stats(1 pgs tid 78 v 608) v1 from osd.1 -1694> 2013-04-03 10:49:24.656773 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29165 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) -1693> 2013-04-03 10:49:24.933867 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394150 encode_pending v394151 -1692> 2013-04-03 10:49:24.933899 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427611) queue_proposal bl 443 bytes; ctx = 0x330b0a0 -1691> 2013-04-03 10:49:24.933906 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427611) propose_queued 427612 443 bytes -1690> 2013-04-03 10:49:24.935147 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427611 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579280 -1689> 2013-04-03 10:49:24.935180 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427611 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579000 -1688> 2013-04-03 10:49:24.935260 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29165 encode_pending v 29166 -1687> 2013-04-03 10:49:24.935290 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427577..427611) queue_proposal bl 646 bytes; ctx = 0x1b91a30 -1686> 2013-04-03 10:49:24.935296 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427577..427611) propose_new_value not active; proposal queued -1685> 2013-04-03 10:49:24.962298 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 58 ==== paxos(accept lc 427611 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3930573200 0 0) 0x3579280 con 0x208a9a0 -1684> 2013-04-03 10:49:24.962589 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427612 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579500 -1683> 2013-04-03 10:49:24.962617 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427612 fc 0 pn 17800 opn 0) v3 -- ?+0 0x356b000 -1682> 2013-04-03 10:49:25.060815 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 194 ==== paxos(accept lc 427611 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1656916925 0 0) 0x356b000 con 0x208a840 -1681> 2013-04-03 10:49:25.060830 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427612 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3579280 -1680> 2013-04-03 10:49:25.060839 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427612 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3579a00 -1679> 2013-04-03 10:49:25.060850 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos -1678> 2013-04-03 10:49:25.060881 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos version 394151 summary v 394150 -1677> 2013-04-03 10:49:25.060895 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos latest full 393798 -1676> 2013-04-03 10:49:25.060913 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394151 update_from_paxos applying incremental log 394151 2013-04-03 10:49:23.952148 mon.0 10.252.24.25:6789/0 248 : [INF] pgmap v29165: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1225B/s wr, 0op/s -1675> 2013-04-03 10:49:25.060956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 check_subs -1674> 2013-04-03 10:49:25.060992 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 create_pending v 394152 -1673> 2013-04-03 10:49:25.061014 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394151 _updated_log for mon.0 10.252.24.25:6789/0 -1672> 2013-04-03 10:49:25.061027 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 248) v1 -- ?+0 0x33061c0 con 0x1bac160 -1671> 2013-04-03 10:49:25.061052 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427612) propose_queued 427613 646 bytes -1670> 2013-04-03 10:49:25.062097 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427612 fc 0 pn 17800 opn 0) v3 -- ?+0 0x356b280 -1669> 2013-04-03 10:49:25.062128 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427612 fc 0 pn 17800 opn 0) v3 -- ?+0 0x356b500 -1668> 2013-04-03 10:49:25.062155 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 248) v1 ==== 0+0+0 (0 0 0) 0x33061c0 con 0x1bac160 -1667> 2013-04-03 10:49:25.062163 7fcdffcdb700 10 handle_log_ack log(last 248) v1 -1666> 2013-04-03 10:49:25.062164 7fcdffcdb700 10 logged 2013-04-03 10:49:23.952148 mon.0 10.252.24.25:6789/0 248 : [INF] pgmap v29165: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 1225B/s wr, 0op/s -1665> 2013-04-03 10:49:25.062172 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 59 ==== paxos(lease_ack lc 427612 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (214664506 0 0) 0x3579280 con 0x208a9a0 -1664> 2013-04-03 10:49:25.085577 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 195 ==== paxos(lease_ack lc 427612 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (951869202 0 0) 0x356b500 con 0x208a840 -1663> 2013-04-03 10:49:25.086571 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 60 ==== paxos(accept lc 427612 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2096584681 0 0) 0x356b280 con 0x208a9a0 -1662> 2013-04-03 10:49:25.086881 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427613 fc 0 pn 17800 opn 0) v3 -- ?+0 0x356b500 -1661> 2013-04-03 10:49:25.086905 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427613 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579280 -1660> 2013-04-03 10:49:25.110333 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 196 ==== paxos(accept lc 427612 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (334850961 0 0) 0x3579280 con 0x208a840 -1659> 2013-04-03 10:49:25.110346 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427613 fc 427577 pn 0 opn 0) v3 -- ?+0 0x356b280 -1658> 2013-04-03 10:49:25.110354 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427613 fc 427577 pn 0 opn 0) v3 -- ?+0 0x356b000 -1657> 2013-04-03 10:49:25.110395 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29165 update_from_paxos applying incremental 29166 -1656> 2013-04-03 10:49:25.110413 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 v29166: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s -1655> 2013-04-03 10:49:25.110432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 send_pg_creates to 0 pgs -1654> 2013-04-03 10:49:25.110435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 update_logger -1653> 2013-04-03 10:49:25.110444 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 create_pending v 29167 -1652> 2013-04-03 10:49:25.110453 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29166 _updated_stats for osd.1 10.252.24.25:6805/3572 -1651> 2013-04-03 10:49:25.110458 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(1 pgs tid 78) v1 -- ?+0 0x3306540 con 0x3311160 -1650> 2013-04-03 10:49:25.110468 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 check_osd_map already seen 608 >= 608 -1649> 2013-04-03 10:49:25.110470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 update_logger -1648> 2013-04-03 10:49:25.110480 7fcdffcdb700 0 log [INF] : pgmap v29166: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s -1647> 2013-04-03 10:49:25.110481 7fcdffcdb700 10 send_log to self -1646> 2013-04-03 10:49:25.110482 7fcdffcdb700 10 log_queue is 1 last_log 249 sent 248 num 1 unsent 1 sending 1 -1645> 2013-04-03 10:49:25.110484 7fcdffcdb700 10 will send 2013-04-03 10:49:25.110481 mon.0 10.252.24.25:6789/0 249 : [INF] pgmap v29166: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s -1644> 2013-04-03 10:49:25.110490 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2544000 -1643> 2013-04-03 10:49:25.110537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos -1642> 2013-04-03 10:49:25.110554 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos version 394151 summary v 394151 -1641> 2013-04-03 10:49:25.110571 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1640> 2013-04-03 10:49:25.110595 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2544000 con 0x1bac160 -1639> 2013-04-03 10:49:25.110617 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos -1638> 2013-04-03 10:49:25.110632 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos version 394151 summary v 394151 -1637> 2013-04-03 10:49:25.110640 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1636> 2013-04-03 10:49:25.110651 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 preprocess_log log(1 entries) v1 from mon.0 -1635> 2013-04-03 10:49:25.110669 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1634> 2013-04-03 10:49:25.110680 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 prepare_log log(1 entries) v1 from mon.0 -1633> 2013-04-03 10:49:25.110689 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 logging 2013-04-03 10:49:25.110481 mon.0 10.252.24.25:6789/0 249 : [INF] pgmap v29166: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s -1632> 2013-04-03 10:49:25.111581 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 61 ==== paxos(lease_ack lc 427613 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2783223219 0 0) 0x356b280 con 0x208a9a0 -1631> 2013-04-03 10:49:25.111617 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos -1630> 2013-04-03 10:49:25.111633 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos version 394151 summary v 394151 -1629> 2013-04-03 10:49:25.111648 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1628> 2013-04-03 10:49:25.122053 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 197 ==== paxos(lease_ack lc 427613 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (3082235449 0 0) 0x356b000 con 0x208a840 -1627> 2013-04-03 10:49:25.122086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos -1626> 2013-04-03 10:49:25.122102 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394151 update_from_paxos version 394151 summary v 394151 -1625> 2013-04-03 10:49:25.122117 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1624> 2013-04-03 10:49:25.122289 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 198 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 239 v188) v2) to leader v1 ==== 560+0+0 (2780632154 0 0) 0x273a000 con 0x208a840 -1623> 2013-04-03 10:49:25.122303 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -1622> 2013-04-03 10:49:25.122312 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a51000 from 10.252.24.88:6789/0 -1621> 2013-04-03 10:49:25.122344 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 239 v188) v2 from mds.0 10.252.24.25:6802/3144 -1620> 2013-04-03 10:49:25.122354 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 239 v188) v2 tid 46) v2 -- ?+0 0x34d9000 con 0x208a840 -1619> 2013-04-03 10:49:25.439688 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 37 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 30 v188) v2 ==== 289+0+0 (2809894285 0 0) 0x273a2c0 con 0x3311000 -1618> 2013-04-03 10:49:25.439743 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 30 v188) v2 from mds.? 10.252.24.88:6800/18040 -1617> 2013-04-03 10:49:25.439753 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 30 v188) v2 -- ?+0 0x273a000 con 0x3311000 -1616> 2013-04-03 10:49:26.086768 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394151 encode_pending v394152 -1615> 2013-04-03 10:49:26.086800 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427613) queue_proposal bl 442 bytes; ctx = 0x330b030 -1614> 2013-04-03 10:49:26.086807 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427613) propose_queued 427614 442 bytes -1613> 2013-04-03 10:49:26.088057 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427613 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594c80 -1612> 2013-04-03 10:49:26.088091 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427613 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594a00 -1611> 2013-04-03 10:49:26.112526 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 199 ==== paxos(accept lc 427613 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1598580432 0 0) 0x3594a00 con 0x208a840 -1610> 2013-04-03 10:49:26.123983 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427614 fc 0 pn 17800 opn 0) v3 -- ?+0 0x356b000 -1609> 2013-04-03 10:49:26.124006 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427614 fc 0 pn 17800 opn 0) v3 -- ?+0 0x356b280 -1608> 2013-04-03 10:49:26.124020 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 62 ==== paxos(accept lc 427613 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1121640986 0 0) 0x3594c80 con 0x208a9a0 -1607> 2013-04-03 10:49:26.124030 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427614 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3594a00 -1606> 2013-04-03 10:49:26.124046 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427614 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3579280 -1605> 2013-04-03 10:49:26.124055 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos -1604> 2013-04-03 10:49:26.124082 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394151 -1603> 2013-04-03 10:49:26.124096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos latest full 393798 -1602> 2013-04-03 10:49:26.124113 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394152 update_from_paxos applying incremental log 394152 2013-04-03 10:49:25.110481 mon.0 10.252.24.25:6789/0 249 : [INF] pgmap v29166: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s -1601> 2013-04-03 10:49:26.124156 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 check_subs -1600> 2013-04-03 10:49:26.124192 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 create_pending v 394153 -1599> 2013-04-03 10:49:26.124217 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394152 _updated_log for mon.0 10.252.24.25:6789/0 -1598> 2013-04-03 10:49:26.124230 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 249) v1 -- ?+0 0x33061c0 con 0x1bac160 -1597> 2013-04-03 10:49:26.124287 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos -1596> 2013-04-03 10:49:26.124303 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 -1595> 2013-04-03 10:49:26.124319 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1594> 2013-04-03 10:49:26.124340 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 249) v1 ==== 0+0+0 (0 0 0) 0x33061c0 con 0x1bac160 -1593> 2013-04-03 10:49:26.124348 7fcdffcdb700 10 handle_log_ack log(last 249) v1 -1592> 2013-04-03 10:49:26.124349 7fcdffcdb700 10 logged 2013-04-03 10:49:25.110481 mon.0 10.252.24.25:6789/0 249 : [INF] pgmap v29166: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s -1591> 2013-04-03 10:49:26.132564 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 200 ==== paxos(lease_ack lc 427614 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2792109722 0 0) 0x3579280 con 0x208a840 -1590> 2013-04-03 10:49:26.132609 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos -1589> 2013-04-03 10:49:26.132639 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 -1588> 2013-04-03 10:49:26.132655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1587> 2013-04-03 10:49:26.133394 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 63 ==== paxos(lease_ack lc 427614 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (3218846944 0 0) 0x3594a00 con 0x208a9a0 -1586> 2013-04-03 10:49:26.133434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos -1585> 2013-04-03 10:49:26.133450 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 -1584> 2013-04-03 10:49:26.133467 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1583> 2013-04-03 10:49:27.604355 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 243 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140525 v188) v2 ==== 289+0+0 (1251081546 0 0) 0x2cc5600 con 0x2118420 -1582> 2013-04-03 10:49:27.604432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140525 v188) v2 from mds.? 10.252.58.31:6800/10908 -1581> 2013-04-03 10:49:27.604444 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140525 v188) v2 -- ?+0 0x273a2c0 con 0x2118420 -1580> 2013-04-03 10:49:28.393901 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29166 check_down_pgs -1579> 2013-04-03 10:49:28.393925 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29166 v29166: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 534B/s wr, 0op/s -1578> 2013-04-03 10:49:28.393953 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -1577> 2013-04-03 10:49:28.393977 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -1576> 2013-04-03 10:49:28.394013 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -1575> 2013-04-03 10:49:28.394044 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos -1574> 2013-04-03 10:49:28.394063 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 -1573> 2013-04-03 10:49:28.394072 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394152 log -1572> 2013-04-03 10:49:28.394089 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1571> 2013-04-03 10:49:28.394111 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -1570> 2013-04-03 10:49:28.951156 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 201 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 240 v188) v2) to leader v1 ==== 560+0+0 (1435054768 0 0) 0x2cc5340 con 0x208a840 -1569> 2013-04-03 10:49:28.951178 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -1568> 2013-04-03 10:49:28.951192 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a52600 from 10.252.24.88:6789/0 -1567> 2013-04-03 10:49:28.951246 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 240 v188) v2 from mds.0 10.252.24.25:6802/3144 -1566> 2013-04-03 10:49:28.951257 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 240 v188) v2 tid 47) v2 -- ?+0 0x34d9480 con 0x208a840 -1565> 2013-04-03 10:49:29.124101 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427614 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3594780 -1564> 2013-04-03 10:49:29.124134 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427614 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3594500 -1563> 2013-04-03 10:49:29.124727 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 64 ==== paxos(lease_ack lc 427614 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1907853745 0 0) 0x3594780 con 0x208a9a0 -1562> 2013-04-03 10:49:29.124794 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos -1561> 2013-04-03 10:49:29.124815 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 -1560> 2013-04-03 10:49:29.124833 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1559> 2013-04-03 10:49:29.124854 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 202 ==== paxos(lease_ack lc 427614 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (204570446 0 0) 0x3594500 con 0x208a840 -1558> 2013-04-03 10:49:29.124886 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos -1557> 2013-04-03 10:49:29.124902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 -1556> 2013-04-03 10:49:29.125200 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1555> 2013-04-03 10:49:29.439790 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 38 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 31 v188) v2 ==== 289+0+0 (2215979760 0 0) 0x273b600 con 0x3311000 -1554> 2013-04-03 10:49:29.439844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 31 v188) v2 from mds.? 10.252.24.88:6800/18040 -1553> 2013-04-03 10:49:29.439854 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 31 v188) v2 -- ?+0 0x2cc5340 con 0x3311000 -1552> 2013-04-03 10:49:29.657186 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 17 ==== pg_stats(1 pgs tid 79 v 608) v1 ==== 477+0+0 (485510156 0 0) 0x3598900 con 0x3311160 -1551> 2013-04-03 10:49:29.657227 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 preprocess_query pg_stats(1 pgs tid 79 v 608) v1 from osd.1 10.252.24.25:6805/3572 -1550> 2013-04-03 10:49:29.657250 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 prepare_update pg_stats(1 pgs tid 79 v 608) v1 from osd.1 10.252.24.25:6805/3572 -1549> 2013-04-03 10:49:29.657255 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 prepare_pg_stats pg_stats(1 pgs tid 79 v 608) v1 from osd.1 -1548> 2013-04-03 10:49:29.657260 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29166 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) -1547> 2013-04-03 10:49:29.707467 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29166 encode_pending v 29167 -1546> 2013-04-03 10:49:29.707503 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427614) queue_proposal bl 646 bytes; ctx = 0x330b190 -1545> 2013-04-03 10:49:29.707510 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427614) propose_queued 427615 646 bytes -1544> 2013-04-03 10:49:29.709270 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427614 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594280 -1543> 2013-04-03 10:49:29.709310 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427614 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594000 -1542> 2013-04-03 10:49:29.734456 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 203 ==== paxos(accept lc 427614 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3729448530 0 0) 0x3594000 con 0x208a840 -1541> 2013-04-03 10:49:29.735839 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427615 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594500 -1540> 2013-04-03 10:49:29.735869 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427615 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594780 -1539> 2013-04-03 10:49:29.748181 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 65 ==== paxos(accept lc 427614 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (4201107019 0 0) 0x3594500 con 0x208a9a0 -1538> 2013-04-03 10:49:29.748211 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427615 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3594000 -1537> 2013-04-03 10:49:29.748222 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427615 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3594a00 -1536> 2013-04-03 10:49:29.748292 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29166 update_from_paxos applying incremental 29167 -1535> 2013-04-03 10:49:29.748313 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 v29167: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5650B/s wr, 0op/s -1534> 2013-04-03 10:49:29.748331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 send_pg_creates to 0 pgs -1533> 2013-04-03 10:49:29.748334 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 update_logger -1532> 2013-04-03 10:49:29.748344 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 create_pending v 29168 -1531> 2013-04-03 10:49:29.748353 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29167 _updated_stats for osd.1 10.252.24.25:6805/3572 -1530> 2013-04-03 10:49:29.748358 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(1 pgs tid 79) v1 -- ?+0 0x33061c0 con 0x3311160 -1529> 2013-04-03 10:49:29.748369 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 check_osd_map already seen 608 >= 608 -1528> 2013-04-03 10:49:29.748371 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 update_logger -1527> 2013-04-03 10:49:29.748382 7fcdffcdb700 0 log [INF] : pgmap v29167: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5650B/s wr, 0op/s -1526> 2013-04-03 10:49:29.748383 7fcdffcdb700 10 send_log to self -1525> 2013-04-03 10:49:29.748384 7fcdffcdb700 10 log_queue is 1 last_log 250 sent 249 num 1 unsent 1 sending 1 -1524> 2013-04-03 10:49:29.748386 7fcdffcdb700 10 will send 2013-04-03 10:49:29.748383 mon.0 10.252.24.25:6789/0 250 : [INF] pgmap v29167: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5650B/s wr, 0op/s -1523> 2013-04-03 10:49:29.748394 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x25f0200 -1522> 2013-04-03 10:49:29.748447 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos -1521> 2013-04-03 10:49:29.748467 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 -1520> 2013-04-03 10:49:29.748485 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1519> 2013-04-03 10:49:29.748509 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x25f0200 con 0x1bac160 -1518> 2013-04-03 10:49:29.748531 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos -1517> 2013-04-03 10:49:29.748546 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 -1516> 2013-04-03 10:49:29.748554 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1515> 2013-04-03 10:49:29.748564 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 preprocess_log log(1 entries) v1 from mon.0 -1514> 2013-04-03 10:49:29.748588 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1513> 2013-04-03 10:49:29.748600 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 prepare_log log(1 entries) v1 from mon.0 -1512> 2013-04-03 10:49:29.748610 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 logging 2013-04-03 10:49:29.748383 mon.0 10.252.24.25:6789/0 250 : [INF] pgmap v29167: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5650B/s wr, 0op/s -1511> 2013-04-03 10:49:29.759663 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 66 ==== paxos(lease_ack lc 427615 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1674613787 0 0) 0x3594000 con 0x208a9a0 -1510> 2013-04-03 10:49:29.759748 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos -1509> 2013-04-03 10:49:29.759768 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 -1508> 2013-04-03 10:49:29.759785 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1507> 2013-04-03 10:49:29.761629 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 204 ==== paxos(lease_ack lc 427615 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1603250515 0 0) 0x3594a00 con 0x208a840 -1506> 2013-04-03 10:49:29.761664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos -1505> 2013-04-03 10:49:29.761680 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394152 update_from_paxos version 394152 summary v 394152 -1504> 2013-04-03 10:49:29.761699 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1503> 2013-04-03 10:49:29.879374 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.0 10.252.58.31:6802/2682 9 ==== pg_stats(1 pgs tid 414 v 608) v1 ==== 485+0+0 (3461254466 0 0) 0x3598d80 con 0x3311580 -1502> 2013-04-03 10:49:29.879420 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 preprocess_query pg_stats(1 pgs tid 414 v 608) v1 from osd.0 10.252.58.31:6802/2682 -1501> 2013-04-03 10:49:29.879443 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 prepare_update pg_stats(1 pgs tid 414 v 608) v1 from osd.0 10.252.58.31:6802/2682 -1500> 2013-04-03 10:49:29.879449 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 prepare_pg_stats pg_stats(1 pgs tid 414 v 608) v1 from osd.0 -1499> 2013-04-03 10:49:29.879456 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 got osd.0 osd_stat(1143 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1142 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -1498> 2013-04-03 10:49:30.303602 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 205 ==== forward(pg_stats(1 pgs tid 61 v 608) v1) to leader v1 ==== 757+0+0 (1722668585 0 0) 0x2cc5600 con 0x208a840 -1497> 2013-04-03 10:49:30.303619 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -1496> 2013-04-03 10:49:30.303628 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34d9480 from 10.252.24.88:6789/0 -1495> 2013-04-03 10:49:30.303660 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 preprocess_query pg_stats(1 pgs tid 61 v 608) v1 from osd.2 10.252.24.30:6801/5656 -1494> 2013-04-03 10:49:30.303681 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 prepare_update pg_stats(1 pgs tid 61 v 608) v1 from osd.2 10.252.24.30:6801/5656 -1493> 2013-04-03 10:49:30.303686 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 prepare_pg_stats pg_stats(1 pgs tid 61 v 608) v1 from osd.2 -1492> 2013-04-03 10:49:30.303690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29167 got osd.2 osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -1491> 2013-04-03 10:49:30.734655 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29167 encode_pending v 29168 -1490> 2013-04-03 10:49:30.734700 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427615) queue_proposal bl 1093 bytes; ctx = 0x2c3df00 -1489> 2013-04-03 10:49:30.734710 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427615) propose_queued 427616 1093 bytes -1488> 2013-04-03 10:49:30.735801 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427615 fc 0 pn 17800 opn 0) v3 -- ?+0 0x35a9c80 -1487> 2013-04-03 10:49:30.735838 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427615 fc 0 pn 17800 opn 0) v3 -- ?+0 0x35a9a00 -1486> 2013-04-03 10:49:30.735915 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394152 encode_pending v394153 -1485> 2013-04-03 10:49:30.735940 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427577..427615) queue_proposal bl 443 bytes; ctx = 0x2c3d250 -1484> 2013-04-03 10:49:30.735945 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427577..427615) propose_new_value not active; proposal queued -1483> 2013-04-03 10:49:30.760801 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 206 ==== paxos(accept lc 427615 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3914710986 0 0) 0x35a9a00 con 0x208a840 -1482> 2013-04-03 10:49:30.761179 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427616 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594a00 -1481> 2013-04-03 10:49:30.761198 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427616 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594000 -1480> 2013-04-03 10:49:30.766716 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 67 ==== paxos(accept lc 427615 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2510317632 0 0) 0x3594a00 con 0x208a9a0 -1479> 2013-04-03 10:49:30.766749 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427616 fc 427577 pn 0 opn 0) v3 -- ?+0 0x35a9a00 -1478> 2013-04-03 10:49:30.766759 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427616 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3594500 -1477> 2013-04-03 10:49:30.766829 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29167 update_from_paxos applying incremental 29168 -1476> 2013-04-03 10:49:30.766853 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 v29168: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 7415B/s wr, 1op/s -1475> 2013-04-03 10:49:30.766872 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 send_pg_creates to 0 pgs -1474> 2013-04-03 10:49:30.766874 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 update_logger -1473> 2013-04-03 10:49:30.766885 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 create_pending v 29169 -1472> 2013-04-03 10:49:30.766894 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29168 _updated_stats for osd.0 10.252.58.31:6802/2682 -1471> 2013-04-03 10:49:30.766899 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6802/2682 -- pg_stats_ack(1 pgs tid 414) v1 -- ?+0 0x336b380 con 0x3311580 -1470> 2013-04-03 10:49:30.766909 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29168 _updated_stats for osd.2 10.252.24.30:6801/5656 -1469> 2013-04-03 10:49:30.766913 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 61) v1 tid 48) v2 -- ?+0 0x3598d80 con 0x208a840 -1468> 2013-04-03 10:49:30.766922 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 check_osd_map already seen 608 >= 608 -1467> 2013-04-03 10:49:30.766924 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 update_logger -1466> 2013-04-03 10:49:30.766934 7fcdffcdb700 0 log [INF] : pgmap v29168: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 7415B/s wr, 1op/s -1465> 2013-04-03 10:49:30.766938 7fcdffcdb700 10 send_log to self -1464> 2013-04-03 10:49:30.766938 7fcdffcdb700 10 log_queue is 2 last_log 251 sent 250 num 2 unsent 1 sending 1 -1463> 2013-04-03 10:49:30.766940 7fcdffcdb700 10 will send 2013-04-03 10:49:29.748383 mon.0 10.252.24.25:6789/0 250 : [INF] pgmap v29167: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5650B/s wr, 0op/s -1462> 2013-04-03 10:49:30.766947 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x2544000 -1461> 2013-04-03 10:49:30.766973 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427616) propose_queued 427617 443 bytes -1460> 2013-04-03 10:49:30.767246 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427616 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579280 -1459> 2013-04-03 10:49:30.767273 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427616 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594c80 -1458> 2013-04-03 10:49:30.767298 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x2544000 con 0x1bac160 -1457> 2013-04-03 10:49:30.778100 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 68 ==== paxos(lease_ack lc 427616 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2719328246 0 0) 0x3579280 con 0x208a9a0 -1456> 2013-04-03 10:49:30.786221 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 69 ==== paxos(accept lc 427616 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1572529204 0 0) 0x3579500 con 0x208a9a0 -1455> 2013-04-03 10:49:30.786468 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427617 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579280 -1454> 2013-04-03 10:49:30.786496 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427617 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594a00 -1453> 2013-04-03 10:49:30.786508 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 207 ==== paxos(lease_ack lc 427616 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2932602415 0 0) 0x3476500 con 0x208a840 -1452> 2013-04-03 10:49:30.798084 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 208 ==== paxos(accept lc 427616 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2674117032 0 0) 0x3594a00 con 0x208a840 -1451> 2013-04-03 10:49:30.798093 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427617 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3476500 -1450> 2013-04-03 10:49:30.798101 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427617 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3579500 -1449> 2013-04-03 10:49:30.798109 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos -1448> 2013-04-03 10:49:30.798133 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394152 -1447> 2013-04-03 10:49:30.798147 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos latest full 393798 -1446> 2013-04-03 10:49:30.798164 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394153 update_from_paxos applying incremental log 394153 2013-04-03 10:49:29.748383 mon.0 10.252.24.25:6789/0 250 : [INF] pgmap v29167: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5650B/s wr, 0op/s -1445> 2013-04-03 10:49:30.798208 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 check_subs -1444> 2013-04-03 10:49:30.798246 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 create_pending v 394154 -1443> 2013-04-03 10:49:30.798270 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394153 _updated_log for mon.0 10.252.24.25:6789/0 -1442> 2013-04-03 10:49:30.798281 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 250) v1 -- ?+0 0x336b1c0 con 0x1bac160 -1441> 2013-04-03 10:49:30.798301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos -1440> 2013-04-03 10:49:30.798316 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 -1439> 2013-04-03 10:49:30.798326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1438> 2013-04-03 10:49:30.798338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 preprocess_log log(1 entries) v1 from mon.0 -1437> 2013-04-03 10:49:30.798355 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 nothing new -1436> 2013-04-03 10:49:30.798402 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos -1435> 2013-04-03 10:49:30.798418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 -1434> 2013-04-03 10:49:30.798435 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1433> 2013-04-03 10:49:30.798454 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 250) v1 ==== 0+0+0 (0 0 0) 0x336b1c0 con 0x1bac160 -1432> 2013-04-03 10:49:30.798461 7fcdffcdb700 10 handle_log_ack log(last 250) v1 -1431> 2013-04-03 10:49:30.798464 7fcdffcdb700 10 logged 2013-04-03 10:49:29.748383 mon.0 10.252.24.25:6789/0 250 : [INF] pgmap v29167: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 5650B/s wr, 0op/s -1430> 2013-04-03 10:49:30.811491 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 209 ==== paxos(lease_ack lc 427617 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2635349486 0 0) 0x3579500 con 0x208a840 -1429> 2013-04-03 10:49:30.811525 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos -1428> 2013-04-03 10:49:30.811541 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 -1427> 2013-04-03 10:49:30.811556 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1426> 2013-04-03 10:49:30.816486 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 70 ==== paxos(lease_ack lc 427617 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (2198197627 0 0) 0x3476500 con 0x208a9a0 -1425> 2013-04-03 10:49:30.816526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos -1424> 2013-04-03 10:49:30.816543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 -1423> 2013-04-03 10:49:30.816558 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1422> 2013-04-03 10:49:31.604485 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 244 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140526 v188) v2 ==== 289+0+0 (790772941 0 0) 0x2f79b00 con 0x2118420 -1421> 2013-04-03 10:49:31.604565 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140526 v188) v2 from mds.? 10.252.58.31:6800/10908 -1420> 2013-04-03 10:49:31.604583 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140526 v188) v2 -- ?+0 0x2cc5600 con 0x2118420 -1419> 2013-04-03 10:49:32.951165 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 210 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 241 v188) v2) to leader v1 ==== 560+0+0 (13535562 0 0) 0x2a52340 con 0x208a840 -1418> 2013-04-03 10:49:32.951185 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -1417> 2013-04-03 10:49:32.951195 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2146b80 from 10.252.24.88:6789/0 -1416> 2013-04-03 10:49:32.951243 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 241 v188) v2 from mds.0 10.252.24.25:6802/3144 -1415> 2013-04-03 10:49:32.951255 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 241 v188) v2 tid 49) v2 -- ?+0 0x34d9480 con 0x208a840 -1414> 2013-04-03 10:49:33.394237 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29168 check_down_pgs -1413> 2013-04-03 10:49:33.394269 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29168 v29168: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 7415B/s wr, 1op/s -1412> 2013-04-03 10:49:33.394293 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -1411> 2013-04-03 10:49:33.394318 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -1410> 2013-04-03 10:49:33.394355 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -1409> 2013-04-03 10:49:33.394386 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos -1408> 2013-04-03 10:49:33.394405 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 -1407> 2013-04-03 10:49:33.394414 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394153 log -1406> 2013-04-03 10:49:33.394431 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1405> 2013-04-03 10:49:33.395087 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -1404> 2013-04-03 10:49:33.439891 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 39 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 32 v188) v2 ==== 289+0+0 (902870476 0 0) 0x273bb80 con 0x3311000 -1403> 2013-04-03 10:49:33.439940 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 32 v188) v2 from mds.? 10.252.24.88:6800/18040 -1402> 2013-04-03 10:49:33.439949 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 32 v188) v2 -- ?+0 0x2a52340 con 0x3311000 -1401> 2013-04-03 10:49:33.798165 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427617 fc 427577 pn 0 opn 0) v3 -- ?+0 0x35a9780 -1400> 2013-04-03 10:49:33.798199 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427617 fc 427577 pn 0 opn 0) v3 -- ?+0 0x35a9500 -1399> 2013-04-03 10:49:33.798774 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 71 ==== paxos(lease_ack lc 427617 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (3530226232 0 0) 0x35a9780 con 0x208a9a0 -1398> 2013-04-03 10:49:33.798850 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos -1397> 2013-04-03 10:49:33.798871 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 -1396> 2013-04-03 10:49:33.798889 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1395> 2013-04-03 10:49:33.798977 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 211 ==== paxos(lease_ack lc 427617 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1030352391 0 0) 0x35a9500 con 0x208a840 -1394> 2013-04-03 10:49:33.799029 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos -1393> 2013-04-03 10:49:33.799048 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 -1392> 2013-04-03 10:49:33.799064 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1391> 2013-04-03 10:49:34.572659 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 212 ==== forward(pg_stats(0 pgs tid 65 v 608) v1) to leader v1 ==== 368+0+0 (772388619 0 0) 0x2f79b00 con 0x208a840 -1390> 2013-04-03 10:49:34.572681 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.4 10.252.24.32:6800/13473 via mon.2 10.252.24.88:6789/0 -1389> 2013-04-03 10:49:34.572693 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34d9480 from 10.252.24.88:6789/0 -1388> 2013-04-03 10:49:34.572745 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 preprocess_query pg_stats(0 pgs tid 65 v 608) v1 from osd.4 10.252.24.32:6800/13473 -1387> 2013-04-03 10:49:34.572769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 prepare_update pg_stats(0 pgs tid 65 v 608) v1 from osd.4 10.252.24.32:6800/13473 -1386> 2013-04-03 10:49:34.572775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 prepare_pg_stats pg_stats(0 pgs tid 65 v 608) v1 from osd.4 -1385> 2013-04-03 10:49:34.572781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29168 got osd.4 osd_stat(1049 MB used, 56607 MB avail, 59728 MB total, peers []/[]) (was osd_stat(1048 MB used, 56609 MB avail, 59728 MB total, peers []/[])) -1384> 2013-04-03 10:49:34.622964 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29168 encode_pending v 29169 -1383> 2013-04-03 10:49:34.622992 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427617) queue_proposal bl 265 bytes; ctx = 0x2c3d230 -1382> 2013-04-03 10:49:34.623001 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427617) propose_queued 427618 265 bytes -1381> 2013-04-03 10:49:34.624899 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427617 fc 0 pn 17800 opn 0) v3 -- ?+0 0x35a9280 -1380> 2013-04-03 10:49:34.624930 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427617 fc 0 pn 17800 opn 0) v3 -- ?+0 0x35a9000 -1379> 2013-04-03 10:49:34.650674 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 213 ==== paxos(accept lc 427617 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2452128782 0 0) 0x35a9000 con 0x208a840 -1378> 2013-04-03 10:49:34.651014 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427618 fc 0 pn 17800 opn 0) v3 -- ?+0 0x35a9500 -1377> 2013-04-03 10:49:34.651040 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427618 fc 0 pn 17800 opn 0) v3 -- ?+0 0x35a9780 -1376> 2013-04-03 10:49:34.657857 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 18 ==== pg_stats(2 pgs tid 80 v 608) v1 ==== 850+0+0 (164409908 0 0) 0x35986c0 con 0x3311160 -1375> 2013-04-03 10:49:34.857909 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 72 ==== paxos(accept lc 427617 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (4224756192 0 0) 0x35a9500 con 0x208a9a0 -1374> 2013-04-03 10:49:34.857935 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427618 fc 427577 pn 0 opn 0) v3 -- ?+0 0x35a9000 -1373> 2013-04-03 10:49:34.857945 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427618 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3476500 -1372> 2013-04-03 10:49:34.858007 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29168 update_from_paxos applying incremental 29169 -1371> 2013-04-03 10:49:34.858018 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 v29169: 240 pgs: 240 active+clean; 256 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 2916B/s wr, 0op/s -1370> 2013-04-03 10:49:34.858035 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 send_pg_creates to 0 pgs -1369> 2013-04-03 10:49:34.858038 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 update_logger -1368> 2013-04-03 10:49:34.858047 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 create_pending v 29170 -1367> 2013-04-03 10:49:34.858056 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29169 _updated_stats for osd.4 10.252.24.32:6800/13473 -1366> 2013-04-03 10:49:34.858061 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(0 pgs tid 65) v1 tid 50) v2 -- ?+0 0x3598d80 con 0x208a840 -1365> 2013-04-03 10:49:34.858093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 preprocess_query pg_stats(2 pgs tid 80 v 608) v1 from osd.1 10.252.24.25:6805/3572 -1364> 2013-04-03 10:49:34.858118 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 prepare_update pg_stats(2 pgs tid 80 v 608) v1 from osd.1 10.252.24.25:6805/3572 -1363> 2013-04-03 10:49:34.858123 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 prepare_pg_stats pg_stats(2 pgs tid 80 v 608) v1 from osd.1 -1362> 2013-04-03 10:49:34.858129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 got osd.1 osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) -1361> 2013-04-03 10:49:34.858152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 check_osd_map already seen 608 >= 608 -1360> 2013-04-03 10:49:34.858156 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 update_logger -1359> 2013-04-03 10:49:34.858167 7fcdffcdb700 0 log [INF] : pgmap v29169: 240 pgs: 240 active+clean; 256 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 2916B/s wr, 0op/s -1358> 2013-04-03 10:49:34.858169 7fcdffcdb700 10 send_log to self -1357> 2013-04-03 10:49:34.858169 7fcdffcdb700 10 log_queue is 2 last_log 252 sent 250 num 2 unsent 2 sending 2 -1356> 2013-04-03 10:49:34.858171 7fcdffcdb700 10 will send 2013-04-03 10:49:30.766937 mon.0 10.252.24.25:6789/0 251 : [INF] pgmap v29168: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 7415B/s wr, 1op/s -1355> 2013-04-03 10:49:34.858177 7fcdffcdb700 10 will send 2013-04-03 10:49:34.858168 mon.0 10.252.24.25:6789/0 252 : [INF] pgmap v29169: 240 pgs: 240 active+clean; 256 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 2916B/s wr, 0op/s -1354> 2013-04-03 10:49:34.858182 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x33b8e00 -1353> 2013-04-03 10:49:34.858255 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427577..427618) queue_proposal bl 987 bytes; ctx = 0x330b3a0 -1352> 2013-04-03 10:49:34.858265 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427577..427618) propose_queued 427619 987 bytes -1351> 2013-04-03 10:49:34.861754 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427618 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579500 -1350> 2013-04-03 10:49:34.861785 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427618 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594a00 -1349> 2013-04-03 10:49:34.861798 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x33b8e00 con 0x1bac160 -1348> 2013-04-03 10:49:34.861844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos -1347> 2013-04-03 10:49:34.861861 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 -1346> 2013-04-03 10:49:34.861871 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -1345> 2013-04-03 10:49:34.861881 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 preprocess_log log(2 entries) v1 from mon.0 -1344> 2013-04-03 10:49:34.861923 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -1343> 2013-04-03 10:49:34.861938 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 prepare_log log(2 entries) v1 from mon.0 -1342> 2013-04-03 10:49:34.861948 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 logging 2013-04-03 10:49:30.766937 mon.0 10.252.24.25:6789/0 251 : [INF] pgmap v29168: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 7415B/s wr, 1op/s -1341> 2013-04-03 10:49:34.861969 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 logging 2013-04-03 10:49:34.858168 mon.0 10.252.24.25:6789/0 252 : [INF] pgmap v29169: 240 pgs: 240 active+clean; 256 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 2916B/s wr, 0op/s -1340> 2013-04-03 10:49:34.862001 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 214 ==== paxos(lease_ack lc 427618 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (330914235 0 0) 0x3476500 con 0x208a840 -1339> 2013-04-03 10:49:34.879890 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.0 10.252.58.31:6802/2682 10 ==== pg_stats(1 pgs tid 415 v 608) v1 ==== 485+0+0 (3387258202 0 0) 0x3598b40 con 0x3311580 -1338> 2013-04-03 10:49:34.879935 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 preprocess_query pg_stats(1 pgs tid 415 v 608) v1 from osd.0 10.252.58.31:6802/2682 -1337> 2013-04-03 10:49:34.879956 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 prepare_update pg_stats(1 pgs tid 415 v 608) v1 from osd.0 10.252.58.31:6802/2682 -1336> 2013-04-03 10:49:34.879960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 prepare_pg_stats pg_stats(1 pgs tid 415 v 608) v1 from osd.0 -1335> 2013-04-03 10:49:34.879964 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 got osd.0 osd_stat(1143 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1143 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -1334> 2013-04-03 10:49:34.884746 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 215 ==== paxos(accept lc 427618 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (947472486 0 0) 0x3594a00 con 0x208a840 -1333> 2013-04-03 10:49:34.885675 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427619 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3476500 -1332> 2013-04-03 10:49:34.885690 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427619 fc 0 pn 17800 opn 0) v3 -- ?+0 0x35a9500 -1331> 2013-04-03 10:49:34.942455 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 73 ==== paxos(lease_ack lc 427618 fc 427577 pn 0 opn 0) v3 ==== 80+0+0 (1461519235 0 0) 0x3476500 con 0x208a9a0 -1330> 2013-04-03 10:49:34.964236 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 74 ==== paxos(accept lc 427618 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (790740807 0 0) 0x3579500 con 0x208a9a0 -1329> 2013-04-03 10:49:34.964253 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427619 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3476500 -1328> 2013-04-03 10:49:34.964262 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427619 fc 427577 pn 0 opn 0) v3 -- ?+0 0x3594a00 -1327> 2013-04-03 10:49:34.964328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos -1326> 2013-04-03 10:49:34.964346 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 -1325> 2013-04-03 10:49:34.964364 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1324> 2013-04-03 10:49:34.964957 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 216 ==== paxos(lease_ack lc 427619 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (4242150541 0 0) 0x3594a00 con 0x208a840 -1323> 2013-04-03 10:49:34.964990 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos -1322> 2013-04-03 10:49:34.965006 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 -1321> 2013-04-03 10:49:34.965023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1320> 2013-04-03 10:49:35.033163 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 75 ==== paxos(lease_ack lc 427619 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (1565957487 0 0) 0x3476500 con 0x208a9a0 -1319> 2013-04-03 10:49:35.033216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos -1318> 2013-04-03 10:49:35.033234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394153 update_from_paxos version 394153 summary v 394153 -1317> 2013-04-03 10:49:35.033251 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1316> 2013-04-03 10:49:35.304132 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 217 ==== forward(pg_stats(1 pgs tid 62 v 608) v1) to leader v1 ==== 757+0+0 (3304950340 0 0) 0x2145000 con 0x208a840 -1315> 2013-04-03 10:49:35.304151 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -1314> 2013-04-03 10:49:35.304159 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x3598d80 from 10.252.24.88:6789/0 -1313> 2013-04-03 10:49:35.304207 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 preprocess_query pg_stats(1 pgs tid 62 v 608) v1 from osd.2 10.252.24.30:6801/5656 -1312> 2013-04-03 10:49:35.304231 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 prepare_update pg_stats(1 pgs tid 62 v 608) v1 from osd.2 10.252.24.30:6801/5656 -1311> 2013-04-03 10:49:35.304237 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 prepare_pg_stats pg_stats(1 pgs tid 62 v 608) v1 from osd.2 -1310> 2013-04-03 10:49:35.304243 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29169 got osd.2 osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1178 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -1309> 2013-04-03 10:49:35.604592 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 245 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140527 v188) v2 ==== 289+0+0 (206341040 0 0) 0x2f79dc0 con 0x2118420 -1308> 2013-04-03 10:49:35.604661 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140527 v188) v2 from mds.? 10.252.58.31:6800/10908 -1307> 2013-04-03 10:49:35.604673 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140527 v188) v2 -- ?+0 0x2145000 con 0x2118420 -1306> 2013-04-03 10:49:35.650860 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394153 encode_pending v394154 -1305> 2013-04-03 10:49:35.650889 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427608..427619) queue_proposal bl 726 bytes; ctx = 0x330b060 -1304> 2013-04-03 10:49:35.650898 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427619) propose_queued 427620 726 bytes -1303> 2013-04-03 10:49:35.652315 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427619 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594c80 -1302> 2013-04-03 10:49:35.652348 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427619 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594500 -1301> 2013-04-03 10:49:35.652420 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29169 encode_pending v 29170 -1300> 2013-04-03 10:49:35.652458 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427619) queue_proposal bl 1897 bytes; ctx = 0x330b040 -1299> 2013-04-03 10:49:35.652465 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427619) propose_new_value not active; proposal queued -1298> 2013-04-03 10:49:35.676450 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 218 ==== paxos(accept lc 427619 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (4134572996 0 0) 0x3594500 con 0x208a840 -1297> 2013-04-03 10:49:35.682728 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427620 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3476500 -1296> 2013-04-03 10:49:35.682748 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427620 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594a00 -1295> 2013-04-03 10:49:35.693465 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 76 ==== paxos(accept lc 427619 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (4246993679 0 0) 0x3476500 con 0x208a9a0 -1294> 2013-04-03 10:49:35.693491 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427620 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3594500 -1293> 2013-04-03 10:49:35.693501 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427620 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3579500 -1292> 2013-04-03 10:49:35.693513 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos -1291> 2013-04-03 10:49:35.693556 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos version 394154 summary v 394153 -1290> 2013-04-03 10:49:35.693570 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos latest full 393798 -1289> 2013-04-03 10:49:35.693590 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394154 update_from_paxos applying incremental log 394154 2013-04-03 10:49:30.766937 mon.0 10.252.24.25:6789/0 251 : [INF] pgmap v29168: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 7415B/s wr, 1op/s -1288> 2013-04-03 10:49:35.693617 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394154 update_from_paxos applying incremental log 394154 2013-04-03 10:49:34.858168 mon.0 10.252.24.25:6789/0 252 : [INF] pgmap v29169: 240 pgs: 240 active+clean; 256 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 2916B/s wr, 0op/s -1287> 2013-04-03 10:49:35.693655 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 check_subs -1286> 2013-04-03 10:49:35.693691 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 create_pending v 394155 -1285> 2013-04-03 10:49:35.694306 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394154 _updated_log for mon.0 10.252.24.25:6789/0 -1284> 2013-04-03 10:49:35.694321 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 252) v1 -- ?+0 0x336ba80 con 0x1bac160 -1283> 2013-04-03 10:49:35.694348 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427620) propose_queued 427621 1897 bytes -1282> 2013-04-03 10:49:35.695429 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427620 fc 0 pn 17800 opn 0) v3 -- ?+0 0x355a000 -1281> 2013-04-03 10:49:35.695448 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427620 fc 0 pn 17800 opn 0) v3 -- ?+0 0x356ba00 -1280> 2013-04-03 10:49:35.695461 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 252) v1 ==== 0+0+0 (0 0 0) 0x336ba80 con 0x1bac160 -1279> 2013-04-03 10:49:35.695468 7fcdffcdb700 10 handle_log_ack log(last 252) v1 -1278> 2013-04-03 10:49:35.695470 7fcdffcdb700 10 logged 2013-04-03 10:49:30.766937 mon.0 10.252.24.25:6789/0 251 : [INF] pgmap v29168: 240 pgs: 240 active+clean; 256 MB data, 5662 MB used, 1067 GB / 1079 GB avail; 7415B/s wr, 1op/s -1277> 2013-04-03 10:49:35.695477 7fcdffcdb700 10 logged 2013-04-03 10:49:34.858168 mon.0 10.252.24.25:6789/0 252 : [INF] pgmap v29169: 240 pgs: 240 active+clean; 256 MB data, 5663 MB used, 1067 GB / 1079 GB avail; 2916B/s wr, 0op/s -1276> 2013-04-03 10:49:35.699488 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 219 ==== paxos(lease_ack lc 427620 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3684834501 0 0) 0x3579500 con 0x208a840 -1275> 2013-04-03 10:49:35.704058 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 77 ==== paxos(lease_ack lc 427620 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (345665202 0 0) 0x355a000 con 0x208a9a0 -1274> 2013-04-03 10:49:35.722862 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 220 ==== paxos(accept lc 427620 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2812310007 0 0) 0x356ba00 con 0x208a840 -1273> 2013-04-03 10:49:35.723795 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427621 fc 0 pn 17800 opn 0) v3 -- ?+0 0x355a000 -1272> 2013-04-03 10:49:35.723818 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427621 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579500 -1271> 2013-04-03 10:49:35.735113 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 78 ==== paxos(accept lc 427620 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (525325517 0 0) 0x355a000 con 0x208a9a0 -1270> 2013-04-03 10:49:35.735127 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427621 fc 427608 pn 0 opn 0) v3 -- ?+0 0x356ba00 -1269> 2013-04-03 10:49:35.735136 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427621 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3476500 -1268> 2013-04-03 10:49:35.735188 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29169 update_from_paxos applying incremental 29170 -1267> 2013-04-03 10:49:35.735218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 v29170: 240 pgs: 240 active+clean; 256 MB data, 5664 MB used, 1067 GB / 1079 GB avail; 74960B/s wr, 1op/s -1266> 2013-04-03 10:49:35.735239 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 send_pg_creates to 0 pgs -1265> 2013-04-03 10:49:35.735242 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 update_logger -1264> 2013-04-03 10:49:35.735253 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 create_pending v 29171 -1263> 2013-04-03 10:49:35.735262 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29170 _updated_stats for osd.1 10.252.24.25:6805/3572 -1262> 2013-04-03 10:49:35.735269 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(2 pgs tid 80) v1 -- ?+0 0x336b700 con 0x3311160 -1261> 2013-04-03 10:49:35.735278 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29170 _updated_stats for osd.0 10.252.58.31:6802/2682 -1260> 2013-04-03 10:49:35.735283 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6802/2682 -- pg_stats_ack(1 pgs tid 415) v1 -- ?+0 0x336be00 con 0x3311580 -1259> 2013-04-03 10:49:35.735293 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29170 _updated_stats for osd.2 10.252.24.30:6801/5656 -1258> 2013-04-03 10:49:35.735297 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 62) v1 tid 51) v2 -- ?+0 0x3598b40 con 0x208a840 -1257> 2013-04-03 10:49:35.735306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 check_osd_map already seen 608 >= 608 -1256> 2013-04-03 10:49:35.735309 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 update_logger -1255> 2013-04-03 10:49:35.735319 7fcdffcdb700 0 log [INF] : pgmap v29170: 240 pgs: 240 active+clean; 256 MB data, 5664 MB used, 1067 GB / 1079 GB avail; 74960B/s wr, 1op/s -1254> 2013-04-03 10:49:35.735321 7fcdffcdb700 10 send_log to self -1253> 2013-04-03 10:49:35.735322 7fcdffcdb700 10 log_queue is 1 last_log 253 sent 252 num 1 unsent 1 sending 1 -1252> 2013-04-03 10:49:35.735323 7fcdffcdb700 10 will send 2013-04-03 10:49:35.735320 mon.0 10.252.24.25:6789/0 253 : [INF] pgmap v29170: 240 pgs: 240 active+clean; 256 MB data, 5664 MB used, 1067 GB / 1079 GB avail; 74960B/s wr, 1op/s -1251> 2013-04-03 10:49:35.735330 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x33e9600 -1250> 2013-04-03 10:49:35.735379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos -1249> 2013-04-03 10:49:35.735398 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos version 394154 summary v 394154 -1248> 2013-04-03 10:49:35.735417 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1247> 2013-04-03 10:49:35.735436 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x33e9600 con 0x1bac160 -1246> 2013-04-03 10:49:35.735455 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos -1245> 2013-04-03 10:49:35.735471 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos version 394154 summary v 394154 -1244> 2013-04-03 10:49:35.735480 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1243> 2013-04-03 10:49:35.735492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 preprocess_log log(1 entries) v1 from mon.0 -1242> 2013-04-03 10:49:35.735513 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1241> 2013-04-03 10:49:35.735525 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 prepare_log log(1 entries) v1 from mon.0 -1240> 2013-04-03 10:49:35.735535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 logging 2013-04-03 10:49:35.735320 mon.0 10.252.24.25:6789/0 253 : [INF] pgmap v29170: 240 pgs: 240 active+clean; 256 MB data, 5664 MB used, 1067 GB / 1079 GB avail; 74960B/s wr, 1op/s -1239> 2013-04-03 10:49:35.748281 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 221 ==== paxos(lease_ack lc 427621 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3037481778 0 0) 0x3476500 con 0x208a840 -1238> 2013-04-03 10:49:35.748328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos -1237> 2013-04-03 10:49:35.748346 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos version 394154 summary v 394154 -1236> 2013-04-03 10:49:35.748361 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1235> 2013-04-03 10:49:35.760171 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 79 ==== paxos(lease_ack lc 427621 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3123890682 0 0) 0x356ba00 con 0x208a9a0 -1234> 2013-04-03 10:49:35.760218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos -1233> 2013-04-03 10:49:35.760236 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394154 update_from_paxos version 394154 summary v 394154 -1232> 2013-04-03 10:49:35.760252 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1231> 2013-04-03 10:49:36.723069 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394154 encode_pending v394155 -1230> 2013-04-03 10:49:36.723100 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427608..427621) queue_proposal bl 444 bytes; ctx = 0x330b230 -1229> 2013-04-03 10:49:36.723108 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427621) propose_queued 427622 444 bytes -1228> 2013-04-03 10:49:36.724176 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427621 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594000 -1227> 2013-04-03 10:49:36.724210 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427621 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594780 -1226> 2013-04-03 10:49:36.761965 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 80 ==== paxos(accept lc 427621 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (548474430 0 0) 0x3594000 con 0x208a9a0 -1225> 2013-04-03 10:49:36.762931 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427622 fc 0 pn 17800 opn 0) v3 -- ?+0 0x356ba00 -1224> 2013-04-03 10:49:36.762958 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427622 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3476500 -1223> 2013-04-03 10:49:36.764192 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 222 ==== paxos(accept lc 427621 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1765907777 0 0) 0x3476500 con 0x208a840 -1222> 2013-04-03 10:49:36.764209 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427622 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3594000 -1221> 2013-04-03 10:49:36.764218 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427622 fc 427608 pn 0 opn 0) v3 -- ?+0 0x355a000 -1220> 2013-04-03 10:49:36.764228 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos -1219> 2013-04-03 10:49:36.764264 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394154 -1218> 2013-04-03 10:49:36.764278 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos latest full 393798 -1217> 2013-04-03 10:49:36.764296 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394155 update_from_paxos applying incremental log 394155 2013-04-03 10:49:35.735320 mon.0 10.252.24.25:6789/0 253 : [INF] pgmap v29170: 240 pgs: 240 active+clean; 256 MB data, 5664 MB used, 1067 GB / 1079 GB avail; 74960B/s wr, 1op/s -1216> 2013-04-03 10:49:36.764339 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 check_subs -1215> 2013-04-03 10:49:36.764379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 create_pending v 394156 -1214> 2013-04-03 10:49:36.764403 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394155 _updated_log for mon.0 10.252.24.25:6789/0 -1213> 2013-04-03 10:49:36.764416 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 253) v1 -- ?+0 0x336ba80 con 0x1bac160 -1212> 2013-04-03 10:49:36.764479 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos -1211> 2013-04-03 10:49:36.764498 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 -1210> 2013-04-03 10:49:36.764517 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1209> 2013-04-03 10:49:36.764538 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 253) v1 ==== 0+0+0 (0 0 0) 0x336ba80 con 0x1bac160 -1208> 2013-04-03 10:49:36.764546 7fcdffcdb700 10 handle_log_ack log(last 253) v1 -1207> 2013-04-03 10:49:36.764548 7fcdffcdb700 10 logged 2013-04-03 10:49:35.735320 mon.0 10.252.24.25:6789/0 253 : [INF] pgmap v29170: 240 pgs: 240 active+clean; 256 MB data, 5664 MB used, 1067 GB / 1079 GB avail; 74960B/s wr, 1op/s -1206> 2013-04-03 10:49:36.772108 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 81 ==== paxos(lease_ack lc 427622 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3391921872 0 0) 0x3594000 con 0x208a9a0 -1205> 2013-04-03 10:49:36.772155 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos -1204> 2013-04-03 10:49:36.772173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 -1203> 2013-04-03 10:49:36.772188 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1202> 2013-04-03 10:49:36.776218 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 223 ==== paxos(lease_ack lc 427622 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (1723445430 0 0) 0x355a000 con 0x208a840 -1201> 2013-04-03 10:49:36.776261 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos -1200> 2013-04-03 10:49:36.776280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 -1199> 2013-04-03 10:49:36.776296 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1198> 2013-04-03 10:49:36.951224 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 224 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 242 v188) v2) to leader v1 ==== 560+0+0 (3061918482 0 0) 0x2145dc0 con 0x208a840 -1197> 2013-04-03 10:49:36.951246 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -1196> 2013-04-03 10:49:36.951255 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2ce2340 from 10.252.24.88:6789/0 -1195> 2013-04-03 10:49:36.951311 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 242 v188) v2 from mds.0 10.252.24.25:6802/3144 -1194> 2013-04-03 10:49:36.951322 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 242 v188) v2 tid 52) v2 -- ?+0 0x3598d80 con 0x208a840 -1193> 2013-04-03 10:49:37.440059 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 40 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 33 v188) v2 ==== 289+0+0 (381553329 0 0) 0x273adc0 con 0x3311000 -1192> 2013-04-03 10:49:37.440137 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 33 v188) v2 from mds.? 10.252.24.88:6800/18040 -1191> 2013-04-03 10:49:37.440151 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 33 v188) v2 -- ?+0 0x2145dc0 con 0x3311000 -1190> 2013-04-03 10:49:38.395223 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29170 check_down_pgs -1189> 2013-04-03 10:49:38.395256 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29170 v29170: 240 pgs: 240 active+clean; 256 MB data, 5664 MB used, 1067 GB / 1079 GB avail; 74960B/s wr, 1op/s -1188> 2013-04-03 10:49:38.395284 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -1187> 2013-04-03 10:49:38.395308 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -1186> 2013-04-03 10:49:38.395344 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -1185> 2013-04-03 10:49:38.395375 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos -1184> 2013-04-03 10:49:38.395394 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 -1183> 2013-04-03 10:49:38.395403 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394155 log -1182> 2013-04-03 10:49:38.395421 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1181> 2013-04-03 10:49:38.395439 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -1180> 2013-04-03 10:49:38.859452 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 16 ==== pg_stats(2 pgs tid 6 v 608) v1 ==== 858+0+0 (3399314222 0 0) 0x3521000 con 0x33112c0 -1179> 2013-04-03 10:49:38.859806 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 preprocess_query pg_stats(2 pgs tid 6 v 608) v1 from osd.3 10.252.24.88:6804/18376 -1178> 2013-04-03 10:49:38.859836 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 prepare_update pg_stats(2 pgs tid 6 v 608) v1 from osd.3 10.252.24.88:6804/18376 -1177> 2013-04-03 10:49:38.859841 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 prepare_pg_stats pg_stats(2 pgs tid 6 v 608) v1 from osd.3 -1176> 2013-04-03 10:49:38.859847 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29170 got osd.3 osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1184 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -1175> 2013-04-03 10:49:38.910034 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29170 encode_pending v 29171 -1174> 2013-04-03 10:49:38.910480 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427608..427622) queue_proposal bl 1027 bytes; ctx = 0x2c3d260 -1173> 2013-04-03 10:49:38.910492 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427622) propose_queued 427623 1027 bytes -1172> 2013-04-03 10:49:38.912083 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427622 fc 0 pn 17800 opn 0) v3 -- ?+0 0x356b280 -1171> 2013-04-03 10:49:38.912115 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427622 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579a00 -1170> 2013-04-03 10:49:38.938723 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 225 ==== paxos(accept lc 427622 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1193294688 0 0) 0x3579a00 con 0x208a840 -1169> 2013-04-03 10:49:38.957034 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427623 fc 0 pn 17800 opn 0) v3 -- ?+0 0x355a000 -1168> 2013-04-03 10:49:38.957061 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427623 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3594000 -1167> 2013-04-03 10:49:39.063445 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 82 ==== paxos(accept lc 427622 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3454595275 0 0) 0x355a000 con 0x208a9a0 -1166> 2013-04-03 10:49:39.063460 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427623 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3579a00 -1165> 2013-04-03 10:49:39.063469 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427623 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3476500 -1164> 2013-04-03 10:49:39.063512 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29170 update_from_paxos applying incremental 29171 -1163> 2013-04-03 10:49:39.063530 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 v29171: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 86227B/s wr, 1op/s -1162> 2013-04-03 10:49:39.063548 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 send_pg_creates to 0 pgs -1161> 2013-04-03 10:49:39.063552 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 update_logger -1160> 2013-04-03 10:49:39.063563 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 create_pending v 29172 -1159> 2013-04-03 10:49:39.063571 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29171 _updated_stats for osd.3 10.252.24.88:6804/18376 -1158> 2013-04-03 10:49:39.063576 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6804/18376 -- pg_stats_ack(2 pgs tid 6) v1 -- ?+0 0x336ba80 con 0x33112c0 -1157> 2013-04-03 10:49:39.063587 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 check_osd_map already seen 608 >= 608 -1156> 2013-04-03 10:49:39.063589 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 update_logger -1155> 2013-04-03 10:49:39.063599 7fcdffcdb700 0 log [INF] : pgmap v29171: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 86227B/s wr, 1op/s -1154> 2013-04-03 10:49:39.063601 7fcdffcdb700 10 send_log to self -1153> 2013-04-03 10:49:39.063602 7fcdffcdb700 10 log_queue is 1 last_log 254 sent 253 num 1 unsent 1 sending 1 -1152> 2013-04-03 10:49:39.063603 7fcdffcdb700 10 will send 2013-04-03 10:49:39.063600 mon.0 10.252.24.25:6789/0 254 : [INF] pgmap v29171: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 86227B/s wr, 1op/s -1151> 2013-04-03 10:49:39.063610 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x20fba00 -1150> 2013-04-03 10:49:39.063657 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos -1149> 2013-04-03 10:49:39.063677 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 -1148> 2013-04-03 10:49:39.063694 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1147> 2013-04-03 10:49:39.063718 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x20fba00 con 0x1bac160 -1146> 2013-04-03 10:49:39.063754 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos -1145> 2013-04-03 10:49:39.063769 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 -1144> 2013-04-03 10:49:39.063777 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1143> 2013-04-03 10:49:39.063789 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 preprocess_log log(1 entries) v1 from mon.0 -1142> 2013-04-03 10:49:39.063811 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1141> 2013-04-03 10:49:39.063822 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 prepare_log log(1 entries) v1 from mon.0 -1140> 2013-04-03 10:49:39.063831 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 logging 2013-04-03 10:49:39.063600 mon.0 10.252.24.25:6789/0 254 : [INF] pgmap v29171: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 86227B/s wr, 1op/s -1139> 2013-04-03 10:49:39.064216 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 226 ==== paxos(lease_ack lc 427623 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (797526760 0 0) 0x3476500 con 0x208a840 -1138> 2013-04-03 10:49:39.064250 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos -1137> 2013-04-03 10:49:39.064266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 -1136> 2013-04-03 10:49:39.064280 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1135> 2013-04-03 10:49:39.105217 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 83 ==== paxos(lease_ack lc 427623 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (2253786360 0 0) 0x3579a00 con 0x208a9a0 -1134> 2013-04-03 10:49:39.105260 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos -1133> 2013-04-03 10:49:39.105277 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394155 update_from_paxos version 394155 summary v 394155 -1132> 2013-04-03 10:49:39.105292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1131> 2013-04-03 10:49:39.604727 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 246 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140528 v188) v2 ==== 289+0+0 (3354339560 0 0) 0x2ef9b00 con 0x2118420 -1130> 2013-04-03 10:49:39.604778 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140528 v188) v2 from mds.? 10.252.58.31:6800/10908 -1129> 2013-04-03 10:49:39.604789 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140528 v188) v2 -- ?+0 0x273adc0 con 0x2118420 -1128> 2013-04-03 10:49:39.658358 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 19 ==== pg_stats(3 pgs tid 81 v 608) v1 ==== 1223+0+0 (4131501964 0 0) 0x3598480 con 0x3311160 -1127> 2013-04-03 10:49:39.658397 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 preprocess_query pg_stats(3 pgs tid 81 v 608) v1 from osd.1 10.252.24.25:6805/3572 -1126> 2013-04-03 10:49:39.658418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 prepare_update pg_stats(3 pgs tid 81 v 608) v1 from osd.1 10.252.24.25:6805/3572 -1125> 2013-04-03 10:49:39.658422 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 prepare_pg_stats pg_stats(3 pgs tid 81 v 608) v1 from osd.1 -1124> 2013-04-03 10:49:39.658427 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 got osd.1 osd_stat(1104 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1107 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) -1123> 2013-04-03 10:49:39.880465 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.0 10.252.58.31:6802/2682 11 ==== pg_stats(1 pgs tid 416 v 608) v1 ==== 485+0+0 (2145675945 0 0) 0x35e0480 con 0x3311580 -1122> 2013-04-03 10:49:39.880543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 preprocess_query pg_stats(1 pgs tid 416 v 608) v1 from osd.0 10.252.58.31:6802/2682 -1121> 2013-04-03 10:49:39.880568 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 prepare_update pg_stats(1 pgs tid 416 v 608) v1 from osd.0 10.252.58.31:6802/2682 -1120> 2013-04-03 10:49:39.880573 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 prepare_pg_stats pg_stats(1 pgs tid 416 v 608) v1 from osd.0 -1119> 2013-04-03 10:49:39.880579 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29171 got osd.0 osd_stat(1146 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1143 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -1118> 2013-04-03 10:49:39.938912 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394155 encode_pending v394156 -1117> 2013-04-03 10:49:39.938946 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427608..427623) queue_proposal bl 444 bytes; ctx = 0x330b390 -1116> 2013-04-03 10:49:39.938955 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427623) propose_queued 427624 444 bytes -1115> 2013-04-03 10:49:39.940037 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427623 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579000 -1114> 2013-04-03 10:49:39.940073 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427623 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579c80 -1113> 2013-04-03 10:49:39.940136 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29171 encode_pending v 29172 -1112> 2013-04-03 10:49:39.940173 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427623) queue_proposal bl 1831 bytes; ctx = 0x1b91a20 -1111> 2013-04-03 10:49:39.940179 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427623) propose_new_value not active; proposal queued -1110> 2013-04-03 10:49:39.951222 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 84 ==== paxos(accept lc 427623 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2240302077 0 0) 0x3579000 con 0x208a9a0 -1109> 2013-04-03 10:49:39.951546 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427624 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579a00 -1108> 2013-04-03 10:49:39.951572 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427624 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3476500 -1107> 2013-04-03 10:49:39.967695 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 227 ==== paxos(accept lc 427623 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (665880823 0 0) 0x3476500 con 0x208a840 -1106> 2013-04-03 10:49:39.967708 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427624 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3579000 -1105> 2013-04-03 10:49:39.967716 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427624 fc 427608 pn 0 opn 0) v3 -- ?+0 0x355a000 -1104> 2013-04-03 10:49:39.967726 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos -1103> 2013-04-03 10:49:39.967750 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos version 394156 summary v 394155 -1102> 2013-04-03 10:49:39.967764 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos latest full 393798 -1101> 2013-04-03 10:49:39.967782 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394156 update_from_paxos applying incremental log 394156 2013-04-03 10:49:39.063600 mon.0 10.252.24.25:6789/0 254 : [INF] pgmap v29171: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 86227B/s wr, 1op/s -1100> 2013-04-03 10:49:39.967826 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 check_subs -1099> 2013-04-03 10:49:39.967861 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 create_pending v 394157 -1098> 2013-04-03 10:49:39.967883 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394156 _updated_log for mon.0 10.252.24.25:6789/0 -1097> 2013-04-03 10:49:39.967896 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 254) v1 -- ?+0 0x2e8ba80 con 0x1bac160 -1096> 2013-04-03 10:49:39.967920 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427624) propose_queued 427625 1831 bytes -1095> 2013-04-03 10:49:39.970639 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427624 fc 0 pn 17800 opn 0) v3 -- ?+0 0x355a500 -1094> 2013-04-03 10:49:39.970670 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427624 fc 0 pn 17800 opn 0) v3 -- ?+0 0x355aa00 -1093> 2013-04-03 10:49:39.970691 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 254) v1 ==== 0+0+0 (0 0 0) 0x2e8ba80 con 0x1bac160 -1092> 2013-04-03 10:49:39.970697 7fcdffcdb700 10 handle_log_ack log(last 254) v1 -1091> 2013-04-03 10:49:39.970698 7fcdffcdb700 10 logged 2013-04-03 10:49:39.063600 mon.0 10.252.24.25:6789/0 254 : [INF] pgmap v29171: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 86227B/s wr, 1op/s -1090> 2013-04-03 10:49:39.970706 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 85 ==== paxos(lease_ack lc 427624 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (4156801432 0 0) 0x3579000 con 0x208a9a0 -1089> 2013-04-03 10:49:39.979722 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 228 ==== paxos(lease_ack lc 427624 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3044005682 0 0) 0x355aa00 con 0x208a840 -1088> 2013-04-03 10:49:39.999510 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 86 ==== paxos(accept lc 427624 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (28434386 0 0) 0x355a500 con 0x208a9a0 -1087> 2013-04-03 10:49:39.999818 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427625 fc 0 pn 17800 opn 0) v3 -- ?+0 0x355aa00 -1086> 2013-04-03 10:49:39.999844 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427625 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579000 -1085> 2013-04-03 10:49:40.005010 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 229 ==== paxos(accept lc 427624 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1140690049 0 0) 0x3579000 con 0x208a840 -1084> 2013-04-03 10:49:40.005023 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427625 fc 427608 pn 0 opn 0) v3 -- ?+0 0x355a500 -1083> 2013-04-03 10:49:40.005031 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427625 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3476500 -1082> 2013-04-03 10:49:40.005074 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29171 update_from_paxos applying incremental 29172 -1081> 2013-04-03 10:49:40.005096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 v29172: 240 pgs: 240 active+clean; 253 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 21016B/s wr, 3op/s -1080> 2013-04-03 10:49:40.005117 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 send_pg_creates to 0 pgs -1079> 2013-04-03 10:49:40.005119 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 update_logger -1078> 2013-04-03 10:49:40.005129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 create_pending v 29173 -1077> 2013-04-03 10:49:40.005138 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29172 _updated_stats for osd.1 10.252.24.25:6805/3572 -1076> 2013-04-03 10:49:40.005143 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(3 pgs tid 81) v1 -- ?+0 0x2e8be00 con 0x3311160 -1075> 2013-04-03 10:49:40.005161 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29172 _updated_stats for osd.0 10.252.58.31:6802/2682 -1074> 2013-04-03 10:49:40.005165 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6802/2682 -- pg_stats_ack(1 pgs tid 416) v1 -- ?+0 0x2e8bc40 con 0x3311580 -1073> 2013-04-03 10:49:40.005173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 check_osd_map already seen 608 >= 608 -1072> 2013-04-03 10:49:40.005175 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 update_logger -1071> 2013-04-03 10:49:40.005185 7fcdffcdb700 0 log [INF] : pgmap v29172: 240 pgs: 240 active+clean; 253 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 21016B/s wr, 3op/s -1070> 2013-04-03 10:49:40.005187 7fcdffcdb700 10 send_log to self -1069> 2013-04-03 10:49:40.005188 7fcdffcdb700 10 log_queue is 1 last_log 255 sent 254 num 1 unsent 1 sending 1 -1068> 2013-04-03 10:49:40.005189 7fcdffcdb700 10 will send 2013-04-03 10:49:40.005186 mon.0 10.252.24.25:6789/0 255 : [INF] pgmap v29172: 240 pgs: 240 active+clean; 253 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 21016B/s wr, 3op/s -1067> 2013-04-03 10:49:40.005195 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x33e9000 -1066> 2013-04-03 10:49:40.005284 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos -1065> 2013-04-03 10:49:40.005302 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos version 394156 summary v 394156 -1064> 2013-04-03 10:49:40.005319 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1063> 2013-04-03 10:49:40.005343 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x33e9000 con 0x1bac160 -1062> 2013-04-03 10:49:40.005363 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos -1061> 2013-04-03 10:49:40.005378 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos version 394156 summary v 394156 -1060> 2013-04-03 10:49:40.005386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1059> 2013-04-03 10:49:40.005396 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 preprocess_log log(1 entries) v1 from mon.0 -1058> 2013-04-03 10:49:40.005413 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -1057> 2013-04-03 10:49:40.005425 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 prepare_log log(1 entries) v1 from mon.0 -1056> 2013-04-03 10:49:40.005434 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 logging 2013-04-03 10:49:40.005186 mon.0 10.252.24.25:6789/0 255 : [INF] pgmap v29172: 240 pgs: 240 active+clean; 253 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 21016B/s wr, 3op/s -1055> 2013-04-03 10:49:40.009196 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 87 ==== paxos(lease_ack lc 427625 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3659385243 0 0) 0x355a500 con 0x208a9a0 -1054> 2013-04-03 10:49:40.009234 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos -1053> 2013-04-03 10:49:40.009250 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos version 394156 summary v 394156 -1052> 2013-04-03 10:49:40.009265 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1051> 2013-04-03 10:49:40.030462 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 230 ==== paxos(lease_ack lc 427625 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (4076481387 0 0) 0x3476500 con 0x208a840 -1050> 2013-04-03 10:49:40.030505 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos -1049> 2013-04-03 10:49:40.030522 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394156 update_from_paxos version 394156 summary v 394156 -1048> 2013-04-03 10:49:40.030537 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -1047> 2013-04-03 10:49:40.304898 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 231 ==== forward(pg_stats(2 pgs tid 63 v 608) v1) to leader v1 ==== 1130+0+0 (397394416 0 0) 0x2f79dc0 con 0x208a840 -1046> 2013-04-03 10:49:40.304921 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -1045> 2013-04-03 10:49:40.304930 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x3598d80 from 10.252.24.88:6789/0 -1044> 2013-04-03 10:49:40.304980 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 preprocess_query pg_stats(2 pgs tid 63 v 608) v1 from osd.2 10.252.24.30:6801/5656 -1043> 2013-04-03 10:49:40.305003 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 prepare_update pg_stats(2 pgs tid 63 v 608) v1 from osd.2 10.252.24.30:6801/5656 -1042> 2013-04-03 10:49:40.305009 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 prepare_pg_stats pg_stats(2 pgs tid 63 v 608) v1 from osd.2 -1041> 2013-04-03 10:49:40.305015 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29172 got osd.2 osd_stat(1182 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1179 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -1040> 2013-04-03 10:49:40.951384 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 232 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 243 v188) v2) to leader v1 ==== 560+0+0 (1811164743 0 0) 0x2a83b00 con 0x208a840 -1039> 2013-04-03 10:49:40.951405 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -1038> 2013-04-03 10:49:40.951415 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a83dc0 from 10.252.24.88:6789/0 -1037> 2013-04-03 10:49:40.951470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 243 v188) v2 from mds.0 10.252.24.25:6802/3144 -1036> 2013-04-03 10:49:40.951481 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 243 v188) v2 tid 54) v2 -- ?+0 0x35e0480 con 0x208a840 -1035> 2013-04-03 10:49:40.999688 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29172 encode_pending v 29173 -1034> 2013-04-03 10:49:40.999728 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427608..427625) queue_proposal bl 1027 bytes; ctx = 0x2c3d240 -1033> 2013-04-03 10:49:40.999737 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427625) propose_queued 427626 1027 bytes -1032> 2013-04-03 10:49:41.000974 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427625 fc 0 pn 17800 opn 0) v3 -- ?+0 0x354d000 -1031> 2013-04-03 10:49:41.001012 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427625 fc 0 pn 17800 opn 0) v3 -- ?+0 0x355a280 -1030> 2013-04-03 10:49:41.001084 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394156 encode_pending v394157 -1029> 2013-04-03 10:49:41.001109 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427625) queue_proposal bl 444 bytes; ctx = 0x2c3d220 -1028> 2013-04-03 10:49:41.001119 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427625) propose_new_value not active; proposal queued -1027> 2013-04-03 10:49:41.029790 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 233 ==== paxos(accept lc 427625 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3024074817 0 0) 0x355a280 con 0x208a840 -1026> 2013-04-03 10:49:41.030685 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427626 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3476500 -1025> 2013-04-03 10:49:41.030715 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427626 fc 0 pn 17800 opn 0) v3 -- ?+0 0x355a500 -1024> 2013-04-03 10:49:41.034530 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 88 ==== paxos(accept lc 427625 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2092763264 0 0) 0x3476500 con 0x208a9a0 -1023> 2013-04-03 10:49:41.034541 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427626 fc 427608 pn 0 opn 0) v3 -- ?+0 0x355a280 -1022> 2013-04-03 10:49:41.034549 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427626 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3579000 -1021> 2013-04-03 10:49:41.034590 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29172 update_from_paxos applying incremental 29173 -1020> 2013-04-03 10:49:41.034611 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 v29173: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1698KB/s wr, 8op/s -1019> 2013-04-03 10:49:41.034631 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 send_pg_creates to 0 pgs -1018> 2013-04-03 10:49:41.034633 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 update_logger -1017> 2013-04-03 10:49:41.034645 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 create_pending v 29174 -1016> 2013-04-03 10:49:41.034654 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29173 _updated_stats for osd.2 10.252.24.30:6801/5656 -1015> 2013-04-03 10:49:41.034659 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(2 pgs tid 63) v1 tid 53) v2 -- ?+0 0x20fcb40 con 0x208a840 -1014> 2013-04-03 10:49:41.034671 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 check_osd_map already seen 608 >= 608 -1013> 2013-04-03 10:49:41.034675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 update_logger -1012> 2013-04-03 10:49:41.034685 7fcdffcdb700 0 log [INF] : pgmap v29173: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1698KB/s wr, 8op/s -1011> 2013-04-03 10:49:41.034687 7fcdffcdb700 10 send_log to self -1010> 2013-04-03 10:49:41.034688 7fcdffcdb700 10 log_queue is 2 last_log 256 sent 255 num 2 unsent 1 sending 1 -1009> 2013-04-03 10:49:41.034690 7fcdffcdb700 10 will send 2013-04-03 10:49:40.005186 mon.0 10.252.24.25:6789/0 255 : [INF] pgmap v29172: 240 pgs: 240 active+clean; 253 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 21016B/s wr, 3op/s -1008> 2013-04-03 10:49:41.034696 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x1c75600 -1007> 2013-04-03 10:49:41.034720 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427626) propose_queued 427627 444 bytes -1006> 2013-04-03 10:49:41.035013 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427626 fc 0 pn 17800 opn 0) v3 -- ?+0 0x354d780 -1005> 2013-04-03 10:49:41.035043 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427626 fc 0 pn 17800 opn 0) v3 -- ?+0 0x353d780 -1004> 2013-04-03 10:49:41.035056 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x1c75600 con 0x1bac160 -1003> 2013-04-03 10:49:41.055288 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 234 ==== paxos(lease_ack lc 427626 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (2066653026 0 0) 0x353d780 con 0x208a840 -1002> 2013-04-03 10:49:41.059526 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 89 ==== paxos(lease_ack lc 427626 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (2951891739 0 0) 0x354d780 con 0x208a9a0 -1001> 2013-04-03 10:49:41.067055 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 235 ==== paxos(accept lc 427626 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3964834091 0 0) 0x3579000 con 0x208a840 -1000> 2013-04-03 10:49:41.067331 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427627 fc 0 pn 17800 opn 0) v3 -- ?+0 0x354d780 -999> 2013-04-03 10:49:41.067359 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427627 fc 0 pn 17800 opn 0) v3 -- ?+0 0x353d780 -998> 2013-04-03 10:49:41.068931 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 90 ==== paxos(accept lc 427626 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3451977923 0 0) 0x354d780 con 0x208a9a0 -997> 2013-04-03 10:49:41.068941 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427627 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3579000 -996> 2013-04-03 10:49:41.068949 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427627 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3476500 -995> 2013-04-03 10:49:41.068957 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos -994> 2013-04-03 10:49:41.068981 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394156 -993> 2013-04-03 10:49:41.068995 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos latest full 393798 -992> 2013-04-03 10:49:41.069012 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394157 update_from_paxos applying incremental log 394157 2013-04-03 10:49:40.005186 mon.0 10.252.24.25:6789/0 255 : [INF] pgmap v29172: 240 pgs: 240 active+clean; 253 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 21016B/s wr, 3op/s -991> 2013-04-03 10:49:41.069058 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 check_subs -990> 2013-04-03 10:49:41.069093 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 create_pending v 394158 -989> 2013-04-03 10:49:41.069116 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394157 _updated_log for mon.0 10.252.24.25:6789/0 -988> 2013-04-03 10:49:41.069128 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 255) v1 -- ?+0 0x2e8b540 con 0x1bac160 -987> 2013-04-03 10:49:41.069149 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos -986> 2013-04-03 10:49:41.069164 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 -985> 2013-04-03 10:49:41.069173 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -984> 2013-04-03 10:49:41.069184 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 preprocess_log log(1 entries) v1 from mon.0 -983> 2013-04-03 10:49:41.069200 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 nothing new -982> 2013-04-03 10:49:41.069246 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos -981> 2013-04-03 10:49:41.069262 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 -980> 2013-04-03 10:49:41.069281 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -979> 2013-04-03 10:49:41.069302 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 255) v1 ==== 0+0+0 (0 0 0) 0x2e8b540 con 0x1bac160 -978> 2013-04-03 10:49:41.069309 7fcdffcdb700 10 handle_log_ack log(last 255) v1 -977> 2013-04-03 10:49:41.069310 7fcdffcdb700 10 logged 2013-04-03 10:49:40.005186 mon.0 10.252.24.25:6789/0 255 : [INF] pgmap v29172: 240 pgs: 240 active+clean; 253 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 21016B/s wr, 3op/s -976> 2013-04-03 10:49:41.080417 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 236 ==== paxos(lease_ack lc 427627 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (518634940 0 0) 0x3476500 con 0x208a840 -975> 2013-04-03 10:49:41.080458 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos -974> 2013-04-03 10:49:41.080474 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 -973> 2013-04-03 10:49:41.080489 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -972> 2013-04-03 10:49:41.101095 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 91 ==== paxos(lease_ack lc 427627 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (1343231960 0 0) 0x3579000 con 0x208a9a0 -971> 2013-04-03 10:49:41.101136 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos -970> 2013-04-03 10:49:41.101153 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 -969> 2013-04-03 10:49:41.101168 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -968> 2013-04-03 10:49:41.440135 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 41 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 34 v188) v2 ==== 289+0+0 (1930283830 0 0) 0x2ce2b80 con 0x3311000 -967> 2013-04-03 10:49:41.440185 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 34 v188) v2 from mds.? 10.252.24.88:6800/18040 -966> 2013-04-03 10:49:41.440195 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 34 v188) v2 -- ?+0 0x2a83b00 con 0x3311000 -965> 2013-04-03 10:49:41.561188 7fcdfdfcc700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x29b4780 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept sd=24 10.252.24.88:33219/0 -964> 2013-04-03 10:49:41.561208 7fcdfdfcc700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4780 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33219/0) -963> 2013-04-03 10:49:41.561221 7fcdfdfcc700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4780 sd=24 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -962> 2013-04-03 10:49:41.561435 7fcdfdfcc700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x29b4780 sd=24 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -961> 2013-04-03 10:49:43.395574 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29173 check_down_pgs -960> 2013-04-03 10:49:43.395605 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29173 v29173: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1698KB/s wr, 8op/s -959> 2013-04-03 10:49:43.395633 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -958> 2013-04-03 10:49:43.395655 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -957> 2013-04-03 10:49:43.395692 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -956> 2013-04-03 10:49:43.395723 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos -955> 2013-04-03 10:49:43.395741 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 -954> 2013-04-03 10:49:43.395750 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394157 log -953> 2013-04-03 10:49:43.395768 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -952> 2013-04-03 10:49:43.395786 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -951> 2013-04-03 10:49:43.604860 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 247 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140529 v188) v2 ==== 289+0+0 (3833698197 0 0) 0x2ef9000 con 0x2118420 -950> 2013-04-03 10:49:43.604908 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140529 v188) v2 from mds.? 10.252.58.31:6800/10908 -949> 2013-04-03 10:49:43.604919 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140529 v188) v2 -- ?+0 0x2ce2b80 con 0x2118420 -948> 2013-04-03 10:49:44.069015 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427627 fc 427608 pn 0 opn 0) v3 -- ?+0 0x354d500 -947> 2013-04-03 10:49:44.069062 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427627 fc 427608 pn 0 opn 0) v3 -- ?+0 0x354d280 -946> 2013-04-03 10:49:44.069727 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 92 ==== paxos(lease_ack lc 427627 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3974497925 0 0) 0x354d500 con 0x208a9a0 -945> 2013-04-03 10:49:44.069803 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos -944> 2013-04-03 10:49:44.069825 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 -943> 2013-04-03 10:49:44.069842 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -942> 2013-04-03 10:49:44.069861 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 237 ==== paxos(lease_ack lc 427627 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3780659404 0 0) 0x354d280 con 0x208a840 -941> 2013-04-03 10:49:44.069891 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos -940> 2013-04-03 10:49:44.069907 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 -939> 2013-04-03 10:49:44.069923 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -938> 2013-04-03 10:49:44.658971 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 20 ==== pg_stats(1 pgs tid 82 v 608) v1 ==== 477+0+0 (1893610402 0 0) 0x3598240 con 0x3311160 -937> 2013-04-03 10:49:44.659019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 preprocess_query pg_stats(1 pgs tid 82 v 608) v1 from osd.1 10.252.24.25:6805/3572 -936> 2013-04-03 10:49:44.659042 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 prepare_update pg_stats(1 pgs tid 82 v 608) v1 from osd.1 10.252.24.25:6805/3572 -935> 2013-04-03 10:49:44.659046 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 prepare_pg_stats pg_stats(1 pgs tid 82 v 608) v1 from osd.1 -934> 2013-04-03 10:49:44.659052 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29173 got osd.1 osd_stat(1104 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1104 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) -933> 2013-04-03 10:49:44.709255 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29173 encode_pending v 29174 -932> 2013-04-03 10:49:44.709291 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427608..427627) queue_proposal bl 646 bytes; ctx = 0x2c3d210 -931> 2013-04-03 10:49:44.709299 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427627) propose_queued 427628 646 bytes -930> 2013-04-03 10:49:44.710229 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427627 fc 0 pn 17800 opn 0) v3 -- ?+0 0x353d280 -929> 2013-04-03 10:49:44.710250 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427627 fc 0 pn 17800 opn 0) v3 -- ?+0 0x353d000 -928> 2013-04-03 10:49:44.744468 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 238 ==== paxos(accept lc 427627 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (707939868 0 0) 0x353d000 con 0x208a840 -927> 2013-04-03 10:49:44.744778 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427628 fc 0 pn 17800 opn 0) v3 -- ?+0 0x354d280 -926> 2013-04-03 10:49:44.744804 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427628 fc 0 pn 17800 opn 0) v3 -- ?+0 0x354d500 -925> 2013-04-03 10:49:44.752186 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 93 ==== paxos(accept lc 427627 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (600200312 0 0) 0x354d280 con 0x208a9a0 -924> 2013-04-03 10:49:44.752199 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427628 fc 427608 pn 0 opn 0) v3 -- ?+0 0x353d000 -923> 2013-04-03 10:49:44.752208 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427628 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3579000 -922> 2013-04-03 10:49:44.752253 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29173 update_from_paxos applying incremental 29174 -921> 2013-04-03 10:49:44.752272 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 v29174: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 729KB/s wr, 1op/s -920> 2013-04-03 10:49:44.752290 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 send_pg_creates to 0 pgs -919> 2013-04-03 10:49:44.752295 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 update_logger -918> 2013-04-03 10:49:44.752306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 create_pending v 29175 -917> 2013-04-03 10:49:44.752315 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29174 _updated_stats for osd.1 10.252.24.25:6805/3572 -916> 2013-04-03 10:49:44.752320 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(1 pgs tid 82) v1 -- ?+0 0x2e8b540 con 0x3311160 -915> 2013-04-03 10:49:44.752331 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 check_osd_map already seen 608 >= 608 -914> 2013-04-03 10:49:44.752333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 update_logger -913> 2013-04-03 10:49:44.752343 7fcdffcdb700 0 log [INF] : pgmap v29174: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 729KB/s wr, 1op/s -912> 2013-04-03 10:49:44.752344 7fcdffcdb700 10 send_log to self -911> 2013-04-03 10:49:44.752345 7fcdffcdb700 10 log_queue is 2 last_log 257 sent 255 num 2 unsent 2 sending 2 -910> 2013-04-03 10:49:44.752347 7fcdffcdb700 10 will send 2013-04-03 10:49:41.034687 mon.0 10.252.24.25:6789/0 256 : [INF] pgmap v29173: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1698KB/s wr, 8op/s -909> 2013-04-03 10:49:44.752353 7fcdffcdb700 10 will send 2013-04-03 10:49:44.752344 mon.0 10.252.24.25:6789/0 257 : [INF] pgmap v29174: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 729KB/s wr, 1op/s -908> 2013-04-03 10:49:44.752357 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x2227400 -907> 2013-04-03 10:49:44.752408 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos -906> 2013-04-03 10:49:44.752428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 -905> 2013-04-03 10:49:44.752448 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -904> 2013-04-03 10:49:44.752470 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x2227400 con 0x1bac160 -903> 2013-04-03 10:49:44.752492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos -902> 2013-04-03 10:49:44.752507 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 -901> 2013-04-03 10:49:44.752516 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -900> 2013-04-03 10:49:44.752527 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 preprocess_log log(2 entries) v1 from mon.0 -899> 2013-04-03 10:49:44.752555 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -898> 2013-04-03 10:49:44.752567 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 prepare_log log(2 entries) v1 from mon.0 -897> 2013-04-03 10:49:44.752576 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 logging 2013-04-03 10:49:41.034687 mon.0 10.252.24.25:6789/0 256 : [INF] pgmap v29173: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1698KB/s wr, 8op/s -896> 2013-04-03 10:49:44.752596 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 logging 2013-04-03 10:49:44.752344 mon.0 10.252.24.25:6789/0 257 : [INF] pgmap v29174: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 729KB/s wr, 1op/s -895> 2013-04-03 10:49:44.761442 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 94 ==== paxos(lease_ack lc 427628 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3536273173 0 0) 0x353d000 con 0x208a9a0 -894> 2013-04-03 10:49:44.761478 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos -893> 2013-04-03 10:49:44.761494 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 -892> 2013-04-03 10:49:44.761509 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -891> 2013-04-03 10:49:44.770029 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 239 ==== paxos(lease_ack lc 427628 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (328835550 0 0) 0x3579000 con 0x208a840 -890> 2013-04-03 10:49:44.770064 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos -889> 2013-04-03 10:49:44.770080 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394157 update_from_paxos version 394157 summary v 394157 -888> 2013-04-03 10:49:44.770095 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -887> 2013-04-03 10:49:44.951519 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 240 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 244 v188) v2) to leader v1 ==== 560+0+0 (4171035667 0 0) 0x2f79dc0 con 0x208a840 -886> 2013-04-03 10:49:44.951543 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -885> 2013-04-03 10:49:44.951553 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x2a848c0 from 10.252.24.88:6789/0 -884> 2013-04-03 10:49:44.951606 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 244 v188) v2 from mds.0 10.252.24.25:6802/3144 -883> 2013-04-03 10:49:44.951618 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 244 v188) v2 tid 55) v2 -- ?+0 0x3598240 con 0x208a840 -882> 2013-04-03 10:49:45.305043 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 241 ==== forward(pg_stats(1 pgs tid 64 v 608) v1) to leader v1 ==== 757+0+0 (3601773693 0 0) 0x2ef9000 con 0x208a840 -881> 2013-04-03 10:49:45.305057 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -880> 2013-04-03 10:49:45.305063 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x3598240 from 10.252.24.88:6789/0 -879> 2013-04-03 10:49:45.305097 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 preprocess_query pg_stats(1 pgs tid 64 v 608) v1 from osd.2 10.252.24.30:6801/5656 -878> 2013-04-03 10:49:45.305119 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 prepare_update pg_stats(1 pgs tid 64 v 608) v1 from osd.2 10.252.24.30:6801/5656 -877> 2013-04-03 10:49:45.305125 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 prepare_pg_stats pg_stats(1 pgs tid 64 v 608) v1 from osd.2 -876> 2013-04-03 10:49:45.305131 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29174 got osd.2 osd_stat(1182 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1182 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -875> 2013-04-03 10:49:45.440244 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 42 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 35 v188) v2 ==== 289+0+0 (1348695115 0 0) 0x2a84340 con 0x3311000 -874> 2013-04-03 10:49:45.440299 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 35 v188) v2 from mds.? 10.252.24.88:6800/18040 -873> 2013-04-03 10:49:45.440308 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 35 v188) v2 -- ?+0 0x2ef9000 con 0x3311000 -872> 2013-04-03 10:49:45.744667 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29174 encode_pending v 29175 -871> 2013-04-03 10:49:45.744705 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427608..427628) queue_proposal bl 654 bytes; ctx = 0x330b220 -870> 2013-04-03 10:49:45.744713 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427628) propose_queued 427629 654 bytes -869> 2013-04-03 10:49:45.746010 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427628 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3513a00 -868> 2013-04-03 10:49:45.746047 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427628 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3513500 -867> 2013-04-03 10:49:45.746120 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394157 encode_pending v394158 -866> 2013-04-03 10:49:45.746147 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427628) queue_proposal bl 727 bytes; ctx = 0x330b380 -865> 2013-04-03 10:49:45.746152 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427628) propose_new_value not active; proposal queued -864> 2013-04-03 10:49:45.769323 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 242 ==== paxos(accept lc 427628 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (67944778 0 0) 0x3513500 con 0x208a840 -863> 2013-04-03 10:49:45.781973 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427629 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3579000 -862> 2013-04-03 10:49:45.781999 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427629 fc 0 pn 17800 opn 0) v3 -- ?+0 0x353d000 -861> 2013-04-03 10:49:45.787257 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 95 ==== paxos(accept lc 427628 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (104284686 0 0) 0x3579000 con 0x208a9a0 -860> 2013-04-03 10:49:45.787268 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427629 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3513500 -859> 2013-04-03 10:49:45.787278 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427629 fc 427608 pn 0 opn 0) v3 -- ?+0 0x354d280 -858> 2013-04-03 10:49:45.787321 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29174 update_from_paxos applying incremental 29175 -857> 2013-04-03 10:49:45.787340 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 v29175: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9063B/s wr, 1op/s -856> 2013-04-03 10:49:45.787360 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 send_pg_creates to 0 pgs -855> 2013-04-03 10:49:45.787362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 update_logger -854> 2013-04-03 10:49:45.787372 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 create_pending v 29176 -853> 2013-04-03 10:49:45.787381 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29175 _updated_stats for osd.2 10.252.24.30:6801/5656 -852> 2013-04-03 10:49:45.787386 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 64) v1 tid 56) v2 -- ?+0 0x20fcb40 con 0x208a840 -851> 2013-04-03 10:49:45.787401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 check_osd_map already seen 608 >= 608 -850> 2013-04-03 10:49:45.787403 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 update_logger -849> 2013-04-03 10:49:45.787413 7fcdffcdb700 0 log [INF] : pgmap v29175: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9063B/s wr, 1op/s -848> 2013-04-03 10:49:45.787415 7fcdffcdb700 10 send_log to self -847> 2013-04-03 10:49:45.787416 7fcdffcdb700 10 log_queue is 3 last_log 258 sent 257 num 3 unsent 1 sending 1 -846> 2013-04-03 10:49:45.787417 7fcdffcdb700 10 will send 2013-04-03 10:49:44.752344 mon.0 10.252.24.25:6789/0 257 : [INF] pgmap v29174: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 729KB/s wr, 1op/s -845> 2013-04-03 10:49:45.787424 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x25f2000 -844> 2013-04-03 10:49:45.787450 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427629) propose_queued 427630 727 bytes -843> 2013-04-03 10:49:45.790273 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427629 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3476500 -842> 2013-04-03 10:49:45.790300 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427629 fc 0 pn 17800 opn 0) v3 -- ?+0 0x354d780 -841> 2013-04-03 10:49:45.790325 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x25f2000 con 0x1bac160 -840> 2013-04-03 10:49:45.800918 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 243 ==== paxos(lease_ack lc 427629 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3078101624 0 0) 0x354d780 con 0x208a840 -839> 2013-04-03 10:49:45.812250 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 96 ==== paxos(lease_ack lc 427629 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3375731838 0 0) 0x3476500 con 0x208a9a0 -838> 2013-04-03 10:49:45.812803 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 244 ==== paxos(accept lc 427629 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2608931129 0 0) 0x354d280 con 0x208a840 -837> 2013-04-03 10:49:45.813095 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427630 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3476500 -836> 2013-04-03 10:49:45.813125 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427630 fc 0 pn 17800 opn 0) v3 -- ?+0 0x354d780 -835> 2013-04-03 10:49:45.821154 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 97 ==== paxos(accept lc 427629 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2117597618 0 0) 0x3476500 con 0x208a9a0 -834> 2013-04-03 10:49:45.821164 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427630 fc 427608 pn 0 opn 0) v3 -- ?+0 0x354d280 -833> 2013-04-03 10:49:45.821172 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427630 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3579000 -832> 2013-04-03 10:49:45.821180 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos -831> 2013-04-03 10:49:45.821203 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394157 -830> 2013-04-03 10:49:45.821218 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos latest full 393798 -829> 2013-04-03 10:49:45.821234 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394158 update_from_paxos applying incremental log 394158 2013-04-03 10:49:41.034687 mon.0 10.252.24.25:6789/0 256 : [INF] pgmap v29173: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1698KB/s wr, 8op/s -828> 2013-04-03 10:49:45.821260 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394158 update_from_paxos applying incremental log 394158 2013-04-03 10:49:44.752344 mon.0 10.252.24.25:6789/0 257 : [INF] pgmap v29174: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 729KB/s wr, 1op/s -827> 2013-04-03 10:49:45.821296 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 check_subs -826> 2013-04-03 10:49:45.821332 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 create_pending v 394159 -825> 2013-04-03 10:49:45.821354 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394158 _updated_log for mon.0 10.252.24.25:6789/0 -824> 2013-04-03 10:49:45.821366 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 257) v1 -- ?+0 0x28a51c0 con 0x1bac160 -823> 2013-04-03 10:49:45.821386 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos -822> 2013-04-03 10:49:45.821401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 -821> 2013-04-03 10:49:45.821412 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -820> 2013-04-03 10:49:45.821422 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 preprocess_log log(1 entries) v1 from mon.0 -819> 2013-04-03 10:49:45.821439 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 nothing new -818> 2013-04-03 10:49:45.821487 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos -817> 2013-04-03 10:49:45.821502 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 -816> 2013-04-03 10:49:45.821519 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -815> 2013-04-03 10:49:45.821540 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 257) v1 ==== 0+0+0 (0 0 0) 0x28a51c0 con 0x1bac160 -814> 2013-04-03 10:49:45.821548 7fcdffcdb700 10 handle_log_ack log(last 257) v1 -813> 2013-04-03 10:49:45.821550 7fcdffcdb700 10 logged 2013-04-03 10:49:41.034687 mon.0 10.252.24.25:6789/0 256 : [INF] pgmap v29173: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 1698KB/s wr, 8op/s -812> 2013-04-03 10:49:45.821557 7fcdffcdb700 10 logged 2013-04-03 10:49:44.752344 mon.0 10.252.24.25:6789/0 257 : [INF] pgmap v29174: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 729KB/s wr, 1op/s -811> 2013-04-03 10:49:45.829439 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 98 ==== paxos(lease_ack lc 427630 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (2737144017 0 0) 0x354d280 con 0x208a9a0 -810> 2013-04-03 10:49:45.829474 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos -809> 2013-04-03 10:49:45.829490 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 -808> 2013-04-03 10:49:45.829504 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -807> 2013-04-03 10:49:45.838283 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 245 ==== paxos(lease_ack lc 427630 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (1163089057 0 0) 0x3579000 con 0x208a840 -806> 2013-04-03 10:49:45.838322 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos -805> 2013-04-03 10:49:45.838338 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 -804> 2013-04-03 10:49:45.838353 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -803> 2013-04-03 10:49:47.605031 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 248 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140530 v188) v2 ==== 289+0+0 (2167559698 0 0) 0x2efa340 con 0x2118420 -802> 2013-04-03 10:49:47.605108 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140530 v188) v2 from mds.? 10.252.58.31:6800/10908 -801> 2013-04-03 10:49:47.605120 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140530 v188) v2 -- ?+0 0x2a84340 con 0x2118420 -800> 2013-04-03 10:49:48.395939 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29175 check_down_pgs -799> 2013-04-03 10:49:48.395964 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29175 v29175: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9063B/s wr, 1op/s -798> 2013-04-03 10:49:48.395991 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -797> 2013-04-03 10:49:48.396014 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -796> 2013-04-03 10:49:48.396050 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -795> 2013-04-03 10:49:48.396081 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos -794> 2013-04-03 10:49:48.396099 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 -793> 2013-04-03 10:49:48.396108 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394158 log -792> 2013-04-03 10:49:48.396126 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -791> 2013-04-03 10:49:48.396145 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -790> 2013-04-03 10:49:48.821241 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427630 fc 427608 pn 0 opn 0) v3 -- ?+0 0x34e7280 -789> 2013-04-03 10:49:48.821280 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427630 fc 427608 pn 0 opn 0) v3 -- ?+0 0x34e7000 -788> 2013-04-03 10:49:48.821972 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 99 ==== paxos(lease_ack lc 427630 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (4277802838 0 0) 0x34e7280 con 0x208a9a0 -787> 2013-04-03 10:49:48.822046 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos -786> 2013-04-03 10:49:48.822065 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 -785> 2013-04-03 10:49:48.822083 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -784> 2013-04-03 10:49:48.822102 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 246 ==== paxos(lease_ack lc 427630 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3424573472 0 0) 0x34e7000 con 0x208a840 -783> 2013-04-03 10:49:48.822132 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos -782> 2013-04-03 10:49:48.822147 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 -781> 2013-04-03 10:49:48.822162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -780> 2013-04-03 10:49:48.951569 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 247 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 245 v188) v2) to leader v1 ==== 560+0+0 (2066614029 0 0) 0x34adb80 con 0x208a840 -779> 2013-04-03 10:49:48.951584 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -778> 2013-04-03 10:49:48.951592 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34ad8c0 from 10.252.24.88:6789/0 -777> 2013-04-03 10:49:48.951625 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 245 v188) v2 from mds.0 10.252.24.25:6802/3144 -776> 2013-04-03 10:49:48.951636 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 245 v188) v2 tid 57) v2 -- ?+0 0x3598d80 con 0x208a840 -775> 2013-04-03 10:49:49.440380 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 43 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 36 v188) v2 ==== 289+0+0 (3093973048 0 0) 0x3615600 con 0x3311000 -774> 2013-04-03 10:49:49.440458 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 36 v188) v2 from mds.? 10.252.24.88:6800/18040 -773> 2013-04-03 10:49:49.440471 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 36 v188) v2 -- ?+0 0x34adb80 con 0x3311000 -772> 2013-04-03 10:49:51.577200 7fcdfdfcc700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3250500 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept sd=24 10.252.24.88:33220/0 -771> 2013-04-03 10:49:51.577228 7fcdfdfcc700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3250500 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33220/0) -770> 2013-04-03 10:49:51.577243 7fcdfdfcc700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3250500 sd=24 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -769> 2013-04-03 10:49:51.577447 7fcdfdfcc700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3250500 sd=24 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -768> 2013-04-03 10:49:51.605122 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 249 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140531 v188) v2 ==== 289+0+0 (2723998063 0 0) 0x2efa8c0 con 0x2118420 -767> 2013-04-03 10:49:51.605196 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140531 v188) v2 from mds.? 10.252.58.31:6800/10908 -766> 2013-04-03 10:49:51.605209 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140531 v188) v2 -- ?+0 0x3615600 con 0x2118420 -765> 2013-04-03 10:49:51.821321 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427630 fc 427608 pn 0 opn 0) v3 -- ?+0 0x34b2000 -764> 2013-04-03 10:49:51.821365 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427630 fc 427608 pn 0 opn 0) v3 -- ?+0 0x29b4500 -763> 2013-04-03 10:49:51.821985 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 100 ==== paxos(lease_ack lc 427630 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (647347235 0 0) 0x34b2000 con 0x208a9a0 -762> 2013-04-03 10:49:51.822061 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos -761> 2013-04-03 10:49:51.822083 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 -760> 2013-04-03 10:49:51.822101 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -759> 2013-04-03 10:49:51.822121 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 248 ==== paxos(lease_ack lc 427630 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3203096951 0 0) 0x29b4500 con 0x208a840 -758> 2013-04-03 10:49:51.822156 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos -757> 2013-04-03 10:49:51.822172 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 -756> 2013-04-03 10:49:51.822187 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -755> 2013-04-03 10:49:52.951701 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 249 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 246 v188) v2) to leader v1 ==== 560+0+0 (1626629384 0 0) 0x2efa340 con 0x208a840 -754> 2013-04-03 10:49:52.951721 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -753> 2013-04-03 10:49:52.951730 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34ac580 from 10.252.24.88:6789/0 -752> 2013-04-03 10:49:52.951779 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 246 v188) v2 from mds.0 10.252.24.25:6802/3144 -751> 2013-04-03 10:49:52.951791 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 246 v188) v2 tid 58) v2 -- ?+0 0x20fcb40 con 0x208a840 -750> 2013-04-03 10:49:53.396277 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29175 v29175: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9063B/s wr, 1op/s -749> 2013-04-03 10:49:53.396319 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -748> 2013-04-03 10:49:53.396342 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -747> 2013-04-03 10:49:53.396384 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -746> 2013-04-03 10:49:53.396417 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos -745> 2013-04-03 10:49:53.396436 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 -744> 2013-04-03 10:49:53.396445 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394158 log -743> 2013-04-03 10:49:53.396463 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -742> 2013-04-03 10:49:53.396482 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -741> 2013-04-03 10:49:53.440478 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 44 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 37 v188) v2 ==== 289+0+0 (2600793925 0 0) 0x3615340 con 0x3311000 -740> 2013-04-03 10:49:53.440534 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 37 v188) v2 from mds.? 10.252.24.88:6800/18040 -739> 2013-04-03 10:49:53.440546 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 37 v188) v2 -- ?+0 0x2efa340 con 0x3311000 -738> 2013-04-03 10:49:54.660150 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 21 ==== pg_stats(1 pgs tid 83 v 608) v1 ==== 477+0+0 (4188915539 0 0) 0x3598000 con 0x3311160 -737> 2013-04-03 10:49:54.660220 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 preprocess_query pg_stats(1 pgs tid 83 v 608) v1 from osd.1 10.252.24.25:6805/3572 -736> 2013-04-03 10:49:54.660248 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 prepare_update pg_stats(1 pgs tid 83 v 608) v1 from osd.1 10.252.24.25:6805/3572 -735> 2013-04-03 10:49:54.660253 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 prepare_pg_stats pg_stats(1 pgs tid 83 v 608) v1 from osd.1 -734> 2013-04-03 10:49:54.660259 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29175 got osd.1 osd_stat(1104 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1104 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) -733> 2013-04-03 10:49:54.710433 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29175 encode_pending v 29176 -732> 2013-04-03 10:49:54.710473 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427608..427630) queue_proposal bl 646 bytes; ctx = 0x330b210 -731> 2013-04-03 10:49:54.710481 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427630) propose_queued 427631 646 bytes -730> 2013-04-03 10:49:54.711323 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427630 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34e7a00 -729> 2013-04-03 10:49:54.711344 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427630 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34b2500 -728> 2013-04-03 10:49:54.757254 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 250 ==== paxos(accept lc 427630 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2419239918 0 0) 0x34b2500 con 0x208a840 -727> 2013-04-03 10:49:54.757600 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427631 fc 0 pn 17800 opn 0) v3 -- ?+0 0x29b4500 -726> 2013-04-03 10:49:54.757628 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427631 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34b2000 -725> 2013-04-03 10:49:54.763653 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 101 ==== paxos(accept lc 427630 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3412264473 0 0) 0x29b4780 con 0x208a9a0 -724> 2013-04-03 10:49:54.763668 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427631 fc 427608 pn 0 opn 0) v3 -- ?+0 0x34b2500 -723> 2013-04-03 10:49:54.763676 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427631 fc 427608 pn 0 opn 0) v3 -- ?+0 0x34e7000 -722> 2013-04-03 10:49:54.763725 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29175 update_from_paxos applying incremental 29176 -721> 2013-04-03 10:49:54.763744 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 v29176: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2969B/s wr, 0op/s -720> 2013-04-03 10:49:54.763763 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 send_pg_creates to 0 pgs -719> 2013-04-03 10:49:54.763765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 update_logger -718> 2013-04-03 10:49:54.763775 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 create_pending v 29177 -717> 2013-04-03 10:49:54.763786 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29176 _updated_stats for osd.1 10.252.24.25:6805/3572 -716> 2013-04-03 10:49:54.763791 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(1 pgs tid 83) v1 -- ?+0 0x28a51c0 con 0x3311160 -715> 2013-04-03 10:49:54.763801 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 check_osd_map already seen 608 >= 608 -714> 2013-04-03 10:49:54.763803 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 update_logger -713> 2013-04-03 10:49:54.763813 7fcdffcdb700 0 log [INF] : pgmap v29176: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2969B/s wr, 0op/s -712> 2013-04-03 10:49:54.763815 7fcdffcdb700 10 send_log to self -711> 2013-04-03 10:49:54.763816 7fcdffcdb700 10 log_queue is 2 last_log 259 sent 257 num 2 unsent 2 sending 2 -710> 2013-04-03 10:49:54.763817 7fcdffcdb700 10 will send 2013-04-03 10:49:45.787415 mon.0 10.252.24.25:6789/0 258 : [INF] pgmap v29175: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9063B/s wr, 1op/s -709> 2013-04-03 10:49:54.763823 7fcdffcdb700 10 will send 2013-04-03 10:49:54.763814 mon.0 10.252.24.25:6789/0 259 : [INF] pgmap v29176: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2969B/s wr, 0op/s -708> 2013-04-03 10:49:54.763827 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x25f2400 -707> 2013-04-03 10:49:54.763879 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos -706> 2013-04-03 10:49:54.763899 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 -705> 2013-04-03 10:49:54.763917 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -704> 2013-04-03 10:49:54.763938 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x25f2400 con 0x1bac160 -703> 2013-04-03 10:49:54.763960 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos -702> 2013-04-03 10:49:54.763975 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 -701> 2013-04-03 10:49:54.763984 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 preprocess_query log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -700> 2013-04-03 10:49:54.763994 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 preprocess_log log(2 entries) v1 from mon.0 -699> 2013-04-03 10:49:54.764023 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 prepare_update log(2 entries) v1 from mon.0 10.252.24.25:6789/0 -698> 2013-04-03 10:49:54.764037 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 prepare_log log(2 entries) v1 from mon.0 -697> 2013-04-03 10:49:54.764047 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 logging 2013-04-03 10:49:45.787415 mon.0 10.252.24.25:6789/0 258 : [INF] pgmap v29175: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9063B/s wr, 1op/s -696> 2013-04-03 10:49:54.764066 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 logging 2013-04-03 10:49:54.763814 mon.0 10.252.24.25:6789/0 259 : [INF] pgmap v29176: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2969B/s wr, 0op/s -695> 2013-04-03 10:49:54.769427 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 251 ==== paxos(lease_ack lc 427631 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (730678538 0 0) 0x34e7000 con 0x208a840 -694> 2013-04-03 10:49:54.769461 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos -693> 2013-04-03 10:49:54.769477 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 -692> 2013-04-03 10:49:54.769492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -691> 2013-04-03 10:49:54.788657 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 102 ==== paxos(lease_ack lc 427631 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (965815330 0 0) 0x34b2500 con 0x208a9a0 -690> 2013-04-03 10:49:54.788690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos -689> 2013-04-03 10:49:54.788706 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394158 update_from_paxos version 394158 summary v 394158 -688> 2013-04-03 10:49:54.788720 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -687> 2013-04-03 10:49:55.605254 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 250 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140532 v188) v2 ==== 289+0+0 (1247139100 0 0) 0x2efa600 con 0x2118420 -686> 2013-04-03 10:49:55.605341 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140532 v188) v2 from mds.? 10.252.58.31:6800/10908 -685> 2013-04-03 10:49:55.605354 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140532 v188) v2 -- ?+0 0x3615340 con 0x2118420 -684> 2013-04-03 10:49:55.757449 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394158 encode_pending v394159 -683> 2013-04-03 10:49:55.757478 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427608..427631) queue_proposal bl 726 bytes; ctx = 0x2c3d200 -682> 2013-04-03 10:49:55.757485 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427631) propose_queued 427632 726 bytes -681> 2013-04-03 10:49:55.758704 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427631 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34b2280 -680> 2013-04-03 10:49:55.758740 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427631 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3476780 -679> 2013-04-03 10:49:55.785285 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 252 ==== paxos(accept lc 427631 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1090886684 0 0) 0x3476780 con 0x208a840 -678> 2013-04-03 10:49:55.785593 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427632 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34b2500 -677> 2013-04-03 10:49:55.785618 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427632 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34e7000 -676> 2013-04-03 10:49:55.790453 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 103 ==== paxos(accept lc 427631 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2297380865 0 0) 0x34b2500 con 0x208a9a0 -675> 2013-04-03 10:49:55.790465 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427632 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3476780 -674> 2013-04-03 10:49:55.790475 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427632 fc 427608 pn 0 opn 0) v3 -- ?+0 0x29b4780 -673> 2013-04-03 10:49:55.790484 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos -672> 2013-04-03 10:49:55.790511 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394158 -671> 2013-04-03 10:49:55.790526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos latest full 393798 -670> 2013-04-03 10:49:55.790543 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394159 update_from_paxos applying incremental log 394159 2013-04-03 10:49:45.787415 mon.0 10.252.24.25:6789/0 258 : [INF] pgmap v29175: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9063B/s wr, 1op/s -669> 2013-04-03 10:49:55.790569 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394159 update_from_paxos applying incremental log 394159 2013-04-03 10:49:54.763814 mon.0 10.252.24.25:6789/0 259 : [INF] pgmap v29176: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2969B/s wr, 0op/s -668> 2013-04-03 10:49:55.790607 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 check_subs -667> 2013-04-03 10:49:55.790643 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 create_pending v 394160 -666> 2013-04-03 10:49:55.790665 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394159 _updated_log for mon.0 10.252.24.25:6789/0 -665> 2013-04-03 10:49:55.790678 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 259) v1 -- ?+0 0x28a5c40 con 0x1bac160 -664> 2013-04-03 10:49:55.790730 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos -663> 2013-04-03 10:49:55.790746 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 -662> 2013-04-03 10:49:55.790765 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -661> 2013-04-03 10:49:55.790785 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 259) v1 ==== 0+0+0 (0 0 0) 0x28a5c40 con 0x1bac160 -660> 2013-04-03 10:49:55.790792 7fcdffcdb700 10 handle_log_ack log(last 259) v1 -659> 2013-04-03 10:49:55.790793 7fcdffcdb700 10 logged 2013-04-03 10:49:45.787415 mon.0 10.252.24.25:6789/0 258 : [INF] pgmap v29175: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 9063B/s wr, 1op/s -658> 2013-04-03 10:49:55.790798 7fcdffcdb700 10 logged 2013-04-03 10:49:54.763814 mon.0 10.252.24.25:6789/0 259 : [INF] pgmap v29176: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2969B/s wr, 0op/s -657> 2013-04-03 10:49:55.797411 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 104 ==== paxos(lease_ack lc 427632 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (2072945513 0 0) 0x3476780 con 0x208a9a0 -656> 2013-04-03 10:49:55.797446 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos -655> 2013-04-03 10:49:55.797465 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 -654> 2013-04-03 10:49:55.797480 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -653> 2013-04-03 10:49:55.810616 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 253 ==== paxos(lease_ack lc 427632 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (2170500832 0 0) 0x29b4780 con 0x208a840 -652> 2013-04-03 10:49:55.810652 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos -651> 2013-04-03 10:49:55.810668 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 -650> 2013-04-03 10:49:55.810682 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -649> 2013-04-03 10:49:56.951902 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 254 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 247 v188) v2) to leader v1 ==== 560+0+0 (1774041867 0 0) 0x2efa8c0 con 0x208a840 -648> 2013-04-03 10:49:56.951924 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -647> 2013-04-03 10:49:56.951933 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34ac840 from 10.252.24.88:6789/0 -646> 2013-04-03 10:49:56.951989 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 247 v188) v2 from mds.0 10.252.24.25:6802/3144 -645> 2013-04-03 10:49:56.952000 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 247 v188) v2 tid 59) v2 -- ?+0 0x3598d80 con 0x208a840 -644> 2013-04-03 10:49:57.440570 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 45 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 38 v188) v2 ==== 289+0+0 (4273419970 0 0) 0x3615080 con 0x3311000 -643> 2013-04-03 10:49:57.440651 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 38 v188) v2 from mds.? 10.252.24.88:6800/18040 -642> 2013-04-03 10:49:57.440664 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 38 v188) v2 -- ?+0 0x2efa8c0 con 0x3311000 -641> 2013-04-03 10:49:57.563347 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11041 10.252.24.25:0/3745741276 4 ==== mon_subscribe({mdsmap=189+,monmap=6+}) v2 ==== 42+0+0 (3152986420 0 0) 0x3306a80 con 0x319b420 -640> 2013-04-03 10:49:57.563361 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 -639> 2013-04-03 10:49:57.563368 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -638> 2013-04-03 10:49:57.563371 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11041 10.252.24.25:0/3745741276 -- mon_subscribe_ack(300s) v1 -- ?+0 0x28a5c40 -637> 2013-04-03 10:49:57.862799 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11071 10.252.24.32:0/1396772501 4 ==== mon_subscribe({mdsmap=189+,monmap=6+}) v2 ==== 42+0+0 (3152986420 0 0) 0x28a5c40 con 0x3311420 -636> 2013-04-03 10:49:57.862824 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 -635> 2013-04-03 10:49:57.862833 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -634> 2013-04-03 10:49:57.862836 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11071 10.252.24.32:0/1396772501 -- mon_subscribe_ack(300s) v1 -- ?+0 0x3306a80 -633> 2013-04-03 10:49:57.908406 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11074 10.252.24.32:0/3871608469 5 ==== mon_subscribe({mdsmap=189+,monmap=6+}) v2 ==== 42+0+0 (3152986420 0 0) 0x2e8b1c0 con 0x319b2c0 -632> 2013-04-03 10:49:57.908420 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 -631> 2013-04-03 10:49:57.908426 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -630> 2013-04-03 10:49:57.908428 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11074 10.252.24.32:0/3871608469 -- mon_subscribe_ack(300s) v1 -- ?+0 0x28a5c40 -629> 2013-04-03 10:49:58.396622 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29176 check_down_pgs -628> 2013-04-03 10:49:58.396656 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29176 v29176: 240 pgs: 240 active+clean; 256 MB data, 5669 MB used, 1067 GB / 1079 GB avail; 2969B/s wr, 0op/s -627> 2013-04-03 10:49:58.396684 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -626> 2013-04-03 10:49:58.396707 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -625> 2013-04-03 10:49:58.396744 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -624> 2013-04-03 10:49:58.396774 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos -623> 2013-04-03 10:49:58.396793 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 -622> 2013-04-03 10:49:58.396803 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394159 log -621> 2013-04-03 10:49:58.396820 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -620> 2013-04-03 10:49:58.396839 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -619> 2013-04-03 10:49:58.527669 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11051 10.252.24.30:0/802047285 4 ==== mon_subscribe({mdsmap=189+,monmap=6+}) v2 ==== 42+0+0 (3152986420 0 0) 0x3306380 con 0x319b000 -618> 2013-04-03 10:49:58.527686 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 -617> 2013-04-03 10:49:58.527693 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -616> 2013-04-03 10:49:58.527696 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11051 10.252.24.30:0/802047285 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8b1c0 -615> 2013-04-03 10:49:58.790540 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427632 fc 427608 pn 0 opn 0) v3 -- ?+0 0x34b2c80 -614> 2013-04-03 10:49:58.790578 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427632 fc 427608 pn 0 opn 0) v3 -- ?+0 0x348c000 -613> 2013-04-03 10:49:58.791219 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 105 ==== paxos(lease_ack lc 427632 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3107497888 0 0) 0x34b2c80 con 0x208a9a0 -612> 2013-04-03 10:49:58.791287 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos -611> 2013-04-03 10:49:58.791307 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 -610> 2013-04-03 10:49:58.791325 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -609> 2013-04-03 10:49:58.791352 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 255 ==== paxos(lease_ack lc 427632 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (1484493640 0 0) 0x348c000 con 0x208a840 -608> 2013-04-03 10:49:58.791385 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos -607> 2013-04-03 10:49:58.791401 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 -606> 2013-04-03 10:49:58.791418 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -605> 2013-04-03 10:49:59.605397 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 251 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140533 v188) v2 ==== 289+0+0 (1765501537 0 0) 0x2ef9840 con 0x2118420 -604> 2013-04-03 10:49:59.605474 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140533 v188) v2 from mds.? 10.252.58.31:6800/10908 -603> 2013-04-03 10:49:59.605487 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140533 v188) v2 -- ?+0 0x3615080 con 0x2118420 -602> 2013-04-03 10:50:00.306569 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 256 ==== forward(pg_stats(1 pgs tid 65 v 608) v1) to leader v1 ==== 757+0+0 (3251032238 0 0) 0x2efa600 con 0x208a840 -601> 2013-04-03 10:50:00.306590 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from osd.2 10.252.24.30:6801/5656 via mon.2 10.252.24.88:6789/0 -600> 2013-04-03 10:50:00.306600 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x3598d80 from 10.252.24.88:6789/0 -599> 2013-04-03 10:50:00.306651 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 preprocess_query pg_stats(1 pgs tid 65 v 608) v1 from osd.2 10.252.24.30:6801/5656 -598> 2013-04-03 10:50:00.306675 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 prepare_update pg_stats(1 pgs tid 65 v 608) v1 from osd.2 10.252.24.30:6801/5656 -597> 2013-04-03 10:50:00.306681 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 prepare_pg_stats pg_stats(1 pgs tid 65 v 608) v1 from osd.2 -596> 2013-04-03 10:50:00.306687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29176 got osd.2 osd_stat(1182 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[]) (was osd_stat(1182 MB used, 238 GB avail, 241 GB total, peers [0,1,3,4]/[])) -595> 2013-04-03 10:50:00.356834 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29176 encode_pending v 29177 -594> 2013-04-03 10:50:00.356869 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427608..427632) queue_proposal bl 654 bytes; ctx = 0x330b340 -593> 2013-04-03 10:50:00.356878 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427632) propose_queued 427633 654 bytes -592> 2013-04-03 10:50:00.361377 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427632 fc 0 pn 17800 opn 0) v3 -- ?+0 0x348c280 -591> 2013-04-03 10:50:00.361411 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427632 fc 0 pn 17800 opn 0) v3 -- ?+0 0x348c500 -590> 2013-04-03 10:50:00.398572 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 257 ==== paxos(accept lc 427632 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1481772240 0 0) 0x348c500 con 0x208a840 -589> 2013-04-03 10:50:00.406477 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427633 fc 0 pn 17800 opn 0) v3 -- ?+0 0x348c000 -588> 2013-04-03 10:50:00.406505 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427633 fc 0 pn 17800 opn 0) v3 -- ?+0 0x34b2c80 -587> 2013-04-03 10:50:00.406518 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 106 ==== paxos(accept lc 427632 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1403111698 0 0) 0x348c280 con 0x208a9a0 -586> 2013-04-03 10:50:00.406529 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427633 fc 427608 pn 0 opn 0) v3 -- ?+0 0x348c500 -585> 2013-04-03 10:50:00.406546 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427633 fc 427608 pn 0 opn 0) v3 -- ?+0 0x29b4780 -584> 2013-04-03 10:50:00.406861 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29176 update_from_paxos applying incremental 29177 -583> 2013-04-03 10:50:00.406877 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 v29177: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s -582> 2013-04-03 10:50:00.406895 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 send_pg_creates to 0 pgs -581> 2013-04-03 10:50:00.406899 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 update_logger -580> 2013-04-03 10:50:00.406909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 create_pending v 29178 -579> 2013-04-03 10:50:00.406918 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29177 _updated_stats for osd.2 10.252.24.30:6801/5656 -578> 2013-04-03 10:50:00.406923 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(pg_stats_ack(1 pgs tid 65) v1 tid 60) v2 -- ?+0 0x20fcb40 con 0x208a840 -577> 2013-04-03 10:50:00.406934 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 check_osd_map already seen 608 >= 608 -576> 2013-04-03 10:50:00.406936 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 update_logger -575> 2013-04-03 10:50:00.406950 7fcdffcdb700 0 log [INF] : pgmap v29177: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s -574> 2013-04-03 10:50:00.406952 7fcdffcdb700 10 send_log to self -573> 2013-04-03 10:50:00.406953 7fcdffcdb700 10 log_queue is 1 last_log 260 sent 259 num 1 unsent 1 sending 1 -572> 2013-04-03 10:50:00.406954 7fcdffcdb700 10 will send 2013-04-03 10:50:00.406951 mon.0 10.252.24.25:6789/0 260 : [INF] pgmap v29177: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s -571> 2013-04-03 10:50:00.406963 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x25f2400 -570> 2013-04-03 10:50:00.407013 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos -569> 2013-04-03 10:50:00.407032 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 -568> 2013-04-03 10:50:00.407050 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -567> 2013-04-03 10:50:00.407074 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x25f2400 con 0x1bac160 -566> 2013-04-03 10:50:00.407096 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos -565> 2013-04-03 10:50:00.407111 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 -564> 2013-04-03 10:50:00.407119 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -563> 2013-04-03 10:50:00.407129 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 preprocess_log log(1 entries) v1 from mon.0 -562> 2013-04-03 10:50:00.407152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -561> 2013-04-03 10:50:00.407164 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 prepare_log log(1 entries) v1 from mon.0 -560> 2013-04-03 10:50:00.407174 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 logging 2013-04-03 10:50:00.406951 mon.0 10.252.24.25:6789/0 260 : [INF] pgmap v29177: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s -559> 2013-04-03 10:50:00.417245 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 107 ==== paxos(lease_ack lc 427633 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (1074724223 0 0) 0x348c500 con 0x208a9a0 -558> 2013-04-03 10:50:00.417292 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos -557> 2013-04-03 10:50:00.417310 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 -556> 2013-04-03 10:50:00.417326 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -555> 2013-04-03 10:50:00.418889 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 258 ==== paxos(lease_ack lc 427633 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3702488299 0 0) 0x29b4780 con 0x208a840 -554> 2013-04-03 10:50:00.418936 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos -553> 2013-04-03 10:50:00.418954 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394159 update_from_paxos version 394159 summary v 394159 -552> 2013-04-03 10:50:00.418969 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -551> 2013-04-03 10:50:00.847150 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 17 ==== mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=608}) v2 ==== 69+0+0 (83000142 0 0) 0x2e8b8c0 con 0x33112c0 -550> 2013-04-03 10:50:00.847178 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=608}) v2 -549> 2013-04-03 10:50:00.847186 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -548> 2013-04-03 10:50:00.847193 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e608 send_incremental [608..608] to osd.3 10.252.24.88:6804/18376 -547> 2013-04-03 10:50:00.847236 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 build_incremental [608..608] -546> 2013-04-03 10:50:00.847262 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.3 10.252.24.88:6804/18376 -- osd_map(608..608 src has 107..608) v3 -- ?+0 0x33b8400 -545> 2013-04-03 10:50:00.847273 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.3 10.252.24.88:6804/18376 -- mon_subscribe_ack(300s) v1 -- ?+0 0x28a58c0 -544> 2013-04-03 10:50:00.951991 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 259 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 248 v188) v2) to leader v1 ==== 560+0+0 (979418202 0 0) 0x34acb00 con 0x208a840 -543> 2013-04-03 10:50:00.952010 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -542> 2013-04-03 10:50:00.952018 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34ad080 from 10.252.24.88:6789/0 -541> 2013-04-03 10:50:00.952057 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 248 v188) v2 from mds.0 10.252.24.25:6802/3144 -540> 2013-04-03 10:50:00.952068 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 248 v188) v2 tid 61) v2 -- ?+0 0x3598d80 con 0x208a840 -539> 2013-04-03 10:50:01.398770 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394159 encode_pending v394160 -538> 2013-04-03 10:50:01.398797 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427608..427633) queue_proposal bl 443 bytes; ctx = 0x330b3d0 -537> 2013-04-03 10:50:01.398804 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427633) propose_queued 427634 443 bytes -536> 2013-04-03 10:50:01.399858 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427633 fc 0 pn 17800 opn 0) v3 -- ?+0 0x348cc80 -535> 2013-04-03 10:50:01.399899 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427633 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3476000 -534> 2013-04-03 10:50:01.426610 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 260 ==== paxos(accept lc 427633 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (542725164 0 0) 0x3476000 con 0x208a840 -533> 2013-04-03 10:50:01.426908 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427634 fc 0 pn 17800 opn 0) v3 -- ?+0 0x29b4780 -532> 2013-04-03 10:50:01.426934 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427634 fc 0 pn 17800 opn 0) v3 -- ?+0 0x348c500 -531> 2013-04-03 10:50:01.428816 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 108 ==== paxos(accept lc 427633 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3536508171 0 0) 0x29b4780 con 0x208a9a0 -530> 2013-04-03 10:50:01.428827 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427634 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3476000 -529> 2013-04-03 10:50:01.428836 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427634 fc 427608 pn 0 opn 0) v3 -- ?+0 0x348c280 -528> 2013-04-03 10:50:01.428844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos -527> 2013-04-03 10:50:01.428873 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394159 -526> 2013-04-03 10:50:01.428887 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos latest full 393798 -525> 2013-04-03 10:50:01.428904 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394160 update_from_paxos applying incremental log 394160 2013-04-03 10:50:00.406951 mon.0 10.252.24.25:6789/0 260 : [INF] pgmap v29177: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s -524> 2013-04-03 10:50:01.428946 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 check_subs -523> 2013-04-03 10:50:01.428981 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 create_pending v 394161 -522> 2013-04-03 10:50:01.429004 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394160 _updated_log for mon.0 10.252.24.25:6789/0 -521> 2013-04-03 10:50:01.429017 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 260) v1 -- ?+0 0x2e8b8c0 con 0x1bac160 -520> 2013-04-03 10:50:01.429068 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos -519> 2013-04-03 10:50:01.429086 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 -518> 2013-04-03 10:50:01.429104 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -517> 2013-04-03 10:50:01.429125 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 260) v1 ==== 0+0+0 (0 0 0) 0x2e8b8c0 con 0x1bac160 -516> 2013-04-03 10:50:01.429133 7fcdffcdb700 10 handle_log_ack log(last 260) v1 -515> 2013-04-03 10:50:01.429135 7fcdffcdb700 10 logged 2013-04-03 10:50:00.406951 mon.0 10.252.24.25:6789/0 260 : [INF] pgmap v29177: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s -514> 2013-04-03 10:50:01.435507 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 109 ==== paxos(lease_ack lc 427634 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (1658710164 0 0) 0x3476000 con 0x208a9a0 -513> 2013-04-03 10:50:01.435542 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos -512> 2013-04-03 10:50:01.435558 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 -511> 2013-04-03 10:50:01.435572 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -510> 2013-04-03 10:50:01.440640 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 46 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 39 v188) v2 ==== 289+0+0 (3722021311 0 0) 0x3614dc0 con 0x3311000 -509> 2013-04-03 10:50:01.440692 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 39 v188) v2 from mds.? 10.252.24.88:6800/18040 -508> 2013-04-03 10:50:01.440702 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 39 v188) v2 -- ?+0 0x34acb00 con 0x3311000 -507> 2013-04-03 10:50:01.451949 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 261 ==== paxos(lease_ack lc 427634 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (107691472 0 0) 0x348c280 con 0x208a840 -506> 2013-04-03 10:50:01.451987 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos -505> 2013-04-03 10:50:01.452004 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 -504> 2013-04-03 10:50:01.452019 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -503> 2013-04-03 10:50:01.560901 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 22 ==== mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=606}) v2 ==== 69+0+0 (281224682 0 0) 0x22248c0 con 0x3311160 -502> 2013-04-03 10:50:01.560916 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=606}) v2 -501> 2013-04-03 10:50:01.560923 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -500> 2013-04-03 10:50:01.560930 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e608 send_incremental [606..608] to osd.1 10.252.24.25:6805/3572 -499> 2013-04-03 10:50:01.560951 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 build_incremental [606..608] -498> 2013-04-03 10:50:01.560987 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.1 10.252.24.25:6805/3572 -- osd_map(606..608 src has 107..608) v3 -- ?+0 0x25f2400 -497> 2013-04-03 10:50:01.560997 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.1 10.252.24.25:6805/3572 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8b8c0 -496> 2013-04-03 10:50:02.433576 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 47 ==== mon_subscribe({mdsmap=189+,monmap=6+}) v2 ==== 42+0+0 (3152986420 0 0) 0x2e8b1c0 con 0x3311000 -495> 2013-04-03 10:50:02.433603 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 -494> 2013-04-03 10:50:02.433613 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -493> 2013-04-03 10:50:02.433616 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.? 10.252.24.88:6800/18040 -- mon_subscribe_ack(300s) v1 -- ?+0 0x22248c0 -492> 2013-04-03 10:50:03.396982 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29177 check_down_pgs -491> 2013-04-03 10:50:03.397011 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29177 v29177: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1471B/s wr, 0op/s -490> 2013-04-03 10:50:03.397039 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -489> 2013-04-03 10:50:03.397065 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -488> 2013-04-03 10:50:03.397099 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -487> 2013-04-03 10:50:03.397131 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos -486> 2013-04-03 10:50:03.397149 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 -485> 2013-04-03 10:50:03.397157 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394160 log -484> 2013-04-03 10:50:03.397174 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -483> 2013-04-03 10:50:03.397191 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -482> 2013-04-03 10:50:03.605520 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 252 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140534 v188) v2 ==== 289+0+0 (210283494 0 0) 0x20e4b80 con 0x2118420 -481> 2013-04-03 10:50:03.605568 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140534 v188) v2 from mds.? 10.252.58.31:6800/10908 -480> 2013-04-03 10:50:03.605578 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140534 v188) v2 -- ?+0 0x3614dc0 con 0x2118420 -479> 2013-04-03 10:50:04.428905 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427634 fc 427608 pn 0 opn 0) v3 -- ?+0 0x348ca00 -478> 2013-04-03 10:50:04.428953 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427634 fc 427608 pn 0 opn 0) v3 -- ?+0 0x348c780 -477> 2013-04-03 10:50:04.429613 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 110 ==== paxos(lease_ack lc 427634 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (2200744278 0 0) 0x348ca00 con 0x208a9a0 -476> 2013-04-03 10:50:04.429687 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos -475> 2013-04-03 10:50:04.429707 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 -474> 2013-04-03 10:50:04.429725 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -473> 2013-04-03 10:50:04.429749 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 262 ==== paxos(lease_ack lc 427634 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (2226207270 0 0) 0x348c780 con 0x208a840 -472> 2013-04-03 10:50:04.429781 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos -471> 2013-04-03 10:50:04.429797 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 -470> 2013-04-03 10:50:04.429813 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -469> 2013-04-03 10:50:04.661255 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 23 ==== pg_stats(1 pgs tid 84 v 608) v1 ==== 477+0+0 (1263205909 0 0) 0x363fb40 con 0x3311160 -468> 2013-04-03 10:50:04.661300 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 preprocess_query pg_stats(1 pgs tid 84 v 608) v1 from osd.1 10.252.24.25:6805/3572 -467> 2013-04-03 10:50:04.661323 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 prepare_update pg_stats(1 pgs tid 84 v 608) v1 from osd.1 10.252.24.25:6805/3572 -466> 2013-04-03 10:50:04.661328 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 prepare_pg_stats pg_stats(1 pgs tid 84 v 608) v1 from osd.1 -465> 2013-04-03 10:50:04.661334 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29177 got osd.1 osd_stat(1104 MB used, 148 GB avail, 149 GB total, peers [0,4]/[]) (was osd_stat(1104 MB used, 148 GB avail, 149 GB total, peers [0,4]/[])) -464> 2013-04-03 10:50:04.711491 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29177 encode_pending v 29178 -463> 2013-04-03 10:50:04.711523 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427608..427634) queue_proposal bl 646 bytes; ctx = 0x330b330 -462> 2013-04-03 10:50:04.711536 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427634) propose_queued 427635 646 bytes -461> 2013-04-03 10:50:04.714266 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427634 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3645c80 -460> 2013-04-03 10:50:04.714289 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427634 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3645a00 -459> 2013-04-03 10:50:04.748880 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 263 ==== paxos(accept lc 427634 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3653707168 0 0) 0x3645a00 con 0x208a840 -458> 2013-04-03 10:50:04.749292 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427635 fc 0 pn 17800 opn 0) v3 -- ?+0 0x348c780 -457> 2013-04-03 10:50:04.749320 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427635 fc 0 pn 17800 opn 0) v3 -- ?+0 0x348ca00 -456> 2013-04-03 10:50:04.782145 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 111 ==== paxos(accept lc 427634 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1328661779 0 0) 0x348c780 con 0x208a9a0 -455> 2013-04-03 10:50:04.782163 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427635 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3645a00 -454> 2013-04-03 10:50:04.782171 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427635 fc 427608 pn 0 opn 0) v3 -- ?+0 0x348c280 -453> 2013-04-03 10:50:04.782225 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29177 update_from_paxos applying incremental 29178 -452> 2013-04-03 10:50:04.782245 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 v29178: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -451> 2013-04-03 10:50:04.782263 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 send_pg_creates to 0 pgs -450> 2013-04-03 10:50:04.782266 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 update_logger -449> 2013-04-03 10:50:04.782276 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 create_pending v 29179 -448> 2013-04-03 10:50:04.782285 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29178 _updated_stats for osd.1 10.252.24.25:6805/3572 -447> 2013-04-03 10:50:04.782291 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6805/3572 -- pg_stats_ack(1 pgs tid 84) v1 -- ?+0 0x2e8b1c0 con 0x3311160 -446> 2013-04-03 10:50:04.782301 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 check_osd_map already seen 608 >= 608 -445> 2013-04-03 10:50:04.782306 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 update_logger -444> 2013-04-03 10:50:04.782316 7fcdffcdb700 0 log [INF] : pgmap v29178: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -443> 2013-04-03 10:50:04.782318 7fcdffcdb700 10 send_log to self -442> 2013-04-03 10:50:04.782318 7fcdffcdb700 10 log_queue is 1 last_log 261 sent 260 num 1 unsent 1 sending 1 -441> 2013-04-03 10:50:04.782320 7fcdffcdb700 10 will send 2013-04-03 10:50:04.782317 mon.0 10.252.24.25:6789/0 261 : [INF] pgmap v29178: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -440> 2013-04-03 10:50:04.782327 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x25f2a00 -439> 2013-04-03 10:50:04.782379 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos -438> 2013-04-03 10:50:04.782398 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 -437> 2013-04-03 10:50:04.782416 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -436> 2013-04-03 10:50:04.782437 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x25f2a00 con 0x1bac160 -435> 2013-04-03 10:50:04.782459 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos -434> 2013-04-03 10:50:04.782474 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 -433> 2013-04-03 10:50:04.782482 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -432> 2013-04-03 10:50:04.782492 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 preprocess_log log(1 entries) v1 from mon.0 -431> 2013-04-03 10:50:04.782514 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 prepare_update log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -430> 2013-04-03 10:50:04.782526 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 prepare_log log(1 entries) v1 from mon.0 -429> 2013-04-03 10:50:04.782535 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 logging 2013-04-03 10:50:04.782317 mon.0 10.252.24.25:6789/0 261 : [INF] pgmap v29178: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -428> 2013-04-03 10:50:04.782850 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 264 ==== paxos(lease_ack lc 427635 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3607453619 0 0) 0x348c280 con 0x208a840 -427> 2013-04-03 10:50:04.782884 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos -426> 2013-04-03 10:50:04.782902 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 -425> 2013-04-03 10:50:04.782919 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -424> 2013-04-03 10:50:04.807138 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 112 ==== paxos(lease_ack lc 427635 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (679811908 0 0) 0x3645a00 con 0x208a9a0 -423> 2013-04-03 10:50:04.807182 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos -422> 2013-04-03 10:50:04.807199 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394160 update_from_paxos version 394160 summary v 394160 -421> 2013-04-03 10:50:04.807214 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -420> 2013-04-03 10:50:04.883156 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.0 10.252.58.31:6802/2682 12 ==== pg_stats(2 pgs tid 417 v 608) v1 ==== 858+0+0 (3716320413 0 0) 0x35e0240 con 0x3311580 -419> 2013-04-03 10:50:04.883195 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 preprocess_query pg_stats(2 pgs tid 417 v 608) v1 from osd.0 10.252.58.31:6802/2682 -418> 2013-04-03 10:50:04.883216 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 prepare_update pg_stats(2 pgs tid 417 v 608) v1 from osd.0 10.252.58.31:6802/2682 -417> 2013-04-03 10:50:04.883221 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 prepare_pg_stats pg_stats(2 pgs tid 417 v 608) v1 from osd.0 -416> 2013-04-03 10:50:04.883225 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29178 got osd.0 osd_stat(1146 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[]) (was osd_stat(1146 MB used, 268 GB avail, 269 GB total, peers [1,2,3,4]/[])) -415> 2013-04-03 10:50:04.952106 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 265 ==== forward(mdsbeacon(11234/hotcpc9039 up:active seq 249 v188) v2) to leader v1 ==== 560+0+0 (2869030973 0 0) 0x2efa600 con 0x208a840 -414> 2013-04-03 10:50:04.952121 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 received forwarded message from mds.0 10.252.24.25:6802/3144 via mon.2 10.252.24.88:6789/0 -413> 2013-04-03 10:50:04.952130 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 mesg 0x34acdc0 from 10.252.24.88:6789/0 -412> 2013-04-03 10:50:04.952162 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 249 v188) v2 from mds.0 10.252.24.25:6802/3144 -411> 2013-04-03 10:50:04.952173 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6789/0 -- route(mdsbeacon(11234/hotcpc9039 up:active seq 249 v188) v2 tid 62) v2 -- ?+0 0x363fb40 con 0x208a840 -410> 2013-04-03 10:50:05.440792 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 48 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 40 v188) v2 ==== 289+0+0 (726218965 0 0) 0x3614b00 con 0x3311000 -409> 2013-04-03 10:50:05.440864 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11904/hotcpc12533 up:standby seq 40 v188) v2 from mds.? 10.252.24.88:6800/18040 -408> 2013-04-03 10:50:05.440876 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.88:6800/18040 -- mdsbeacon(11904/hotcpc12533 up:standby seq 40 v188) v2 -- ?+0 0x34ac2c0 con 0x3311000 -407> 2013-04-03 10:50:05.749092 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29178 encode_pending v 29179 -406> 2013-04-03 10:50:05.749132 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427608..427635) queue_proposal bl 1027 bytes; ctx = 0x330b370 -405> 2013-04-03 10:50:05.749142 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427635) propose_queued 427636 1027 bytes -404> 2013-04-03 10:50:05.750187 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427635 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3645780 -403> 2013-04-03 10:50:05.750207 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427635 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3645500 -402> 2013-04-03 10:50:05.750368 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394160 encode_pending v394161 -401> 2013-04-03 10:50:05.750400 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427635) queue_proposal bl 443 bytes; ctx = 0x330b360 -400> 2013-04-03 10:50:05.750409 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427635) propose_new_value not active; proposal queued -399> 2013-04-03 10:50:05.775880 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 113 ==== paxos(accept lc 427635 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2375226652 0 0) 0x3645780 con 0x208a9a0 -398> 2013-04-03 10:50:05.776201 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427636 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3645a00 -397> 2013-04-03 10:50:05.776224 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427636 fc 0 pn 17800 opn 0) v3 -- ?+0 0x348c280 -396> 2013-04-03 10:50:05.778180 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 266 ==== paxos(accept lc 427635 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (3979340706 0 0) 0x348c280 con 0x208a840 -395> 2013-04-03 10:50:05.778191 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427636 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3645780 -394> 2013-04-03 10:50:05.778199 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427636 fc 427608 pn 0 opn 0) v3 -- ?+0 0x348c780 -393> 2013-04-03 10:50:05.778244 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29178 update_from_paxos applying incremental 29179 -392> 2013-04-03 10:50:05.778267 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 v29179: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail -391> 2013-04-03 10:50:05.778287 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 send_pg_creates to 0 pgs -390> 2013-04-03 10:50:05.778290 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 update_logger -389> 2013-04-03 10:50:05.778300 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 create_pending v 29180 -388> 2013-04-03 10:50:05.778309 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29179 _updated_stats for osd.0 10.252.58.31:6802/2682 -387> 2013-04-03 10:50:05.778314 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6802/2682 -- pg_stats_ack(2 pgs tid 417) v1 -- ?+0 0x28a5540 con 0x3311580 -386> 2013-04-03 10:50:05.778329 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 check_osd_map already seen 608 >= 608 -385> 2013-04-03 10:50:05.778333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 update_logger -384> 2013-04-03 10:50:05.778344 7fcdffcdb700 0 log [INF] : pgmap v29179: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail -383> 2013-04-03 10:50:05.778345 7fcdffcdb700 10 send_log to self -382> 2013-04-03 10:50:05.778346 7fcdffcdb700 10 log_queue is 2 last_log 262 sent 261 num 2 unsent 1 sending 1 -381> 2013-04-03 10:50:05.778348 7fcdffcdb700 10 will send 2013-04-03 10:50:04.782317 mon.0 10.252.24.25:6789/0 261 : [INF] pgmap v29178: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -380> 2013-04-03 10:50:05.778356 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x20fb400 -379> 2013-04-03 10:50:05.778381 7fcdffcdb700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427636) propose_queued 427637 443 bytes -378> 2013-04-03 10:50:05.789674 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427636 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3476000 -377> 2013-04-03 10:50:05.789702 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427636 fc 0 pn 17800 opn 0) v3 -- ?+0 0x29b4780 -376> 2013-04-03 10:50:05.789726 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x20fb400 con 0x1bac160 -375> 2013-04-03 10:50:05.800871 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 114 ==== paxos(lease_ack lc 427636 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (1442969474 0 0) 0x3476000 con 0x208a9a0 -374> 2013-04-03 10:50:05.803619 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 267 ==== paxos(lease_ack lc 427636 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (3022414992 0 0) 0x29b4780 con 0x208a840 -373> 2013-04-03 10:50:05.806844 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 115 ==== paxos(accept lc 427636 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (1655305586 0 0) 0x3645780 con 0x208a9a0 -372> 2013-04-03 10:50:05.807998 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427637 fc 0 pn 17800 opn 0) v3 -- ?+0 0x29b4780 -371> 2013-04-03 10:50:05.808026 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427637 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3476000 -370> 2013-04-03 10:50:05.815477 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 268 ==== paxos(accept lc 427636 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (330901204 0 0) 0x3476000 con 0x208a840 -369> 2013-04-03 10:50:05.815489 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427637 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3645780 -368> 2013-04-03 10:50:05.815498 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427637 fc 427608 pn 0 opn 0) v3 -- ?+0 0x348c280 -367> 2013-04-03 10:50:05.815506 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos -366> 2013-04-03 10:50:05.815532 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394160 -365> 2013-04-03 10:50:05.815546 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos latest full 393798 -364> 2013-04-03 10:50:05.815563 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394161 update_from_paxos applying incremental log 394161 2013-04-03 10:50:04.782317 mon.0 10.252.24.25:6789/0 261 : [INF] pgmap v29178: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -363> 2013-04-03 10:50:05.815609 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 check_subs -362> 2013-04-03 10:50:05.815645 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 create_pending v 394162 -361> 2013-04-03 10:50:05.815667 7fcdffcdb700 7 mon.hotcpc9039@0(leader).log v394161 _updated_log for mon.0 10.252.24.25:6789/0 -360> 2013-04-03 10:50:05.815679 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6789/0 -- log(last 261) v1 -- ?+0 0x28a5a80 con 0x1bac160 -359> 2013-04-03 10:50:05.815700 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos -358> 2013-04-03 10:50:05.815715 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 -357> 2013-04-03 10:50:05.815724 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 preprocess_query log(1 entries) v1 from mon.0 10.252.24.25:6789/0 -356> 2013-04-03 10:50:05.815735 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 preprocess_log log(1 entries) v1 from mon.0 -355> 2013-04-03 10:50:05.815751 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 nothing new -354> 2013-04-03 10:50:05.815800 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos -353> 2013-04-03 10:50:05.815816 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 -352> 2013-04-03 10:50:05.815832 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -351> 2013-04-03 10:50:05.815853 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(last 261) v1 ==== 0+0+0 (0 0 0) 0x28a5a80 con 0x1bac160 -350> 2013-04-03 10:50:05.815861 7fcdffcdb700 10 handle_log_ack log(last 261) v1 -349> 2013-04-03 10:50:05.815863 7fcdffcdb700 10 logged 2013-04-03 10:50:04.782317 mon.0 10.252.24.25:6789/0 261 : [INF] pgmap v29178: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail; 1638B/s wr, 0op/s -348> 2013-04-03 10:50:05.816088 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 116 ==== paxos(lease_ack lc 427637 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (2361623821 0 0) 0x3645780 con 0x208a9a0 -347> 2013-04-03 10:50:05.816121 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos -346> 2013-04-03 10:50:05.816137 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 -345> 2013-04-03 10:50:05.816152 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -344> 2013-04-03 10:50:05.828920 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.2 10.252.24.88:6789/0 269 ==== paxos(lease_ack lc 427637 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (919460685 0 0) 0x348c280 con 0x208a840 -343> 2013-04-03 10:50:05.828963 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos -342> 2013-04-03 10:50:05.828982 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 -341> 2013-04-03 10:50:05.828997 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -340> 2013-04-03 10:50:06.859536 7fce026be700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=2 pgs=151512 cs=1 l=0).reader couldn't read tag, Success -339> 2013-04-03 10:50:06.859563 7fce026be700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=2 pgs=151512 cs=1 l=0).fault 0: Success -338> 2013-04-03 10:50:06.859579 7fce026be700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=2 pgs=151512 cs=1 l=0).fault with nothing to send, going to standby -337> 2013-04-03 10:50:06.859920 7fcdfdfcc700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3250000 sd=24 :6789 s=0 pgs=0 cs=0 l=0).accept sd=24 10.252.24.25:41806/0 -336> 2013-04-03 10:50:06.859974 7fcdfdecb700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3250a00 sd=30 :6789 s=0 pgs=0 cs=0 l=0).accept sd=30 10.252.24.30:47765/0 -335> 2013-04-03 10:50:06.860001 7fcdfe9d6700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x1f91a00 sd=34 :6789 s=0 pgs=0 cs=0 l=0).accept sd=34 10.252.24.30:47766/0 -334> 2013-04-03 10:50:06.859995 7fcdfd5c2700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x2fe4000 sd=35 :6789 s=0 pgs=0 cs=0 l=0).accept sd=35 10.252.24.25:41808/0 -333> 2013-04-03 10:50:06.860024 7fcdfead7700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x1f91c80 sd=31 :6789 s=0 pgs=0 cs=0 l=0).accept sd=31 10.252.24.25:41807/0 -332> 2013-04-03 10:50:06.860059 7fcdfead7700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.25:6802/3144 mds protocol 0 -331> 2013-04-03 10:50:06.860072 7fcdfd4c1700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3657c80 sd=36 :6789 s=0 pgs=0 cs=0 l=0).accept sd=36 10.252.24.25:41810/0 -330> 2013-04-03 10:50:06.860036 7fcdfd5c2700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.25:0/5246 client protocol 0 -329> 2013-04-03 10:50:06.860097 7fcdfd3c0700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3657a00 sd=37 :6789 s=0 pgs=0 cs=0 l=0).accept sd=37 10.252.24.25:41811/0 -328> 2013-04-03 10:50:06.860168 7fcdfdfcc700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.25:0/1003678 client protocol 0 -327> 2013-04-03 10:50:06.860209 7fcdfd3c0700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.25:0/2614196857 client protocol 0 -326> 2013-04-03 10:50:06.860255 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.0 10.252.24.25:6802/3144 1 ==== auth(proto 0 35 bytes epoch 5) v1 ==== 65+0+0 (3671168286 0 0) 0x24e9400 con 0x3521840 -325> 2013-04-03 10:50:06.860269 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -324> 2013-04-03 10:50:06.860272 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: mds.0 10.252.24.25:6802/3144 is open for mds.0 10.252.24.25:6802/3144 -323> 2013-04-03 10:50:06.860278 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -322> 2013-04-03 10:50:06.860272 7fcdfdecb700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.30:0/517 client protocol 0 -321> 2013-04-03 10:50:06.860318 7fcdfe9d6700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.30:6801/5656 osd protocol 0 -320> 2013-04-03 10:50:06.860322 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -319> 2013-04-03 10:50:06.860342 7fcdfd4c1700 10 mon.hotcpc9039@0(leader) e5 ms_verify_authorizer 10.252.24.25:0/2151250553 client protocol 0 -318> 2013-04-03 10:50:06.860337 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 35 bytes epoch 5) v1 from mds.0 10.252.24.25:6802/3144 -317> 2013-04-03 10:50:06.860350 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=35 -316> 2013-04-03 10:50:06.860383 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6802/3144 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x25f2a00 con 0x3521840 -315> 2013-04-03 10:50:06.860399 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11042 10.252.24.25:0/2614196857 1 ==== auth(proto 0 30 bytes epoch 5) v1 ==== 60+0+0 (1972484708 0 0) 0x24e9e00 con 0x35212c0 -314> 2013-04-03 10:50:06.860405 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -313> 2013-04-03 10:50:06.860407 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11042 10.252.24.25:0/2614196857 is open for client.11042 10.252.24.25:0/2614196857 -312> 2013-04-03 10:50:06.860411 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -311> 2013-04-03 10:50:06.860432 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -310> 2013-04-03 10:50:06.860446 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11042 10.252.24.25:0/2614196857 -309> 2013-04-03 10:50:06.860456 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=30 -308> 2013-04-03 10:50:06.860478 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:0/2614196857 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x20fbc00 con 0x35212c0 -307> 2013-04-03 10:50:06.860506 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11045 10.252.24.25:0/2151250553 1 ==== auth(proto 0 30 bytes epoch 5) v1 ==== 60+0+0 (3821311732 0 0) 0x24e9c00 con 0x3521420 -306> 2013-04-03 10:50:06.860518 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -305> 2013-04-03 10:50:06.860520 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11045 10.252.24.25:0/2151250553 is open for client.11045 10.252.24.25:0/2151250553 -304> 2013-04-03 10:50:06.860524 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -303> 2013-04-03 10:50:06.860551 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -302> 2013-04-03 10:50:06.860579 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11045 10.252.24.25:0/2151250553 -301> 2013-04-03 10:50:06.860592 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=30 -300> 2013-04-03 10:50:06.860615 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:0/2151250553 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x25efe00 con 0x3521420 -299> 2013-04-03 10:50:06.860628 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.? 10.252.24.25:0/5246 1 ==== auth(proto 0 30 bytes epoch 5) v1 ==== 60+0+0 (904349267 0 0) 0x24e9000 con 0x3521580 -298> 2013-04-03 10:50:06.860634 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -297> 2013-04-03 10:50:06.860636 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.? 10.252.24.25:0/5246 is open for client.? 10.252.24.25:0/5246 -296> 2013-04-03 10:50:06.860640 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -295> 2013-04-03 10:50:06.860658 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -294> 2013-04-03 10:50:06.860671 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.? 10.252.24.25:0/5246 -293> 2013-04-03 10:50:06.860683 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=30 -292> 2013-04-03 10:50:06.860703 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:0/5246 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x25f2c00 con 0x3521580 -291> 2013-04-03 10:50:06.860791 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11802 10.252.24.30:0/517 1 ==== auth(proto 0 30 bytes epoch 5) v1 ==== 60+0+0 (300400647 0 0) 0x2b76000 con 0x35219a0 -290> 2013-04-03 10:50:06.860803 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -289> 2013-04-03 10:50:06.860805 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11802 10.252.24.30:0/517 is open for client.11802 10.252.24.30:0/517 -288> 2013-04-03 10:50:06.860809 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -287> 2013-04-03 10:50:06.860830 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -286> 2013-04-03 10:50:06.860844 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 30 bytes epoch 5) v1 from client.11802 10.252.24.30:0/517 -285> 2013-04-03 10:50:06.860859 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=30 -284> 2013-04-03 10:50:06.860881 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:0/517 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x25f2000 con 0x35219a0 -283> 2013-04-03 10:50:06.860894 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.0 10.252.24.25:6802/3144 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (316020240 0 0) 0x287ea00 con 0x3521840 -282> 2013-04-03 10:50:06.860918 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -281> 2013-04-03 10:50:06.860934 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 144 bytes epoch 0) v1 from mds.0 10.252.24.25:6802/3144 -280> 2013-04-03 10:50:06.860944 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=144 -279> 2013-04-03 10:50:06.861082 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6802/3144 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x2b76000 con 0x3521840 -278> 2013-04-03 10:50:06.861098 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.2 10.252.24.30:6801/5656 1 ==== auth(proto 0 26 bytes epoch 5) v1 ==== 56+0+0 (1815740086 0 0) 0x287e000 con 0x35216e0 -277> 2013-04-03 10:50:06.861118 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -276> 2013-04-03 10:50:06.861124 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: osd.2 10.252.24.30:6801/5656 is open for osd.2 10.252.24.30:6801/5656 -275> 2013-04-03 10:50:06.861128 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -274> 2013-04-03 10:50:06.861153 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -273> 2013-04-03 10:50:06.861166 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 26 bytes epoch 5) v1 from osd.2 10.252.24.30:6801/5656 -272> 2013-04-03 10:50:06.861179 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=26 -271> 2013-04-03 10:50:06.861201 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6801/5656 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x2227400 con 0x35216e0 -270> 2013-04-03 10:50:06.861214 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.? 10.252.24.25:0/5246 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (462663606 0 0) 0x25fb400 con 0x3521580 -269> 2013-04-03 10:50:06.861236 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -268> 2013-04-03 10:50:06.861267 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.? 10.252.24.25:0/5246 -267> 2013-04-03 10:50:06.861279 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 -266> 2013-04-03 10:50:06.862180 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:0/5246 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x287e000 con 0x3521580 -265> 2013-04-03 10:50:06.862211 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11045 10.252.24.25:0/2151250553 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (1807490182 0 0) 0x25fb800 con 0x3521420 -264> 2013-04-03 10:50:06.862240 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -263> 2013-04-03 10:50:06.862255 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.11045 10.252.24.25:0/2151250553 -262> 2013-04-03 10:50:06.862265 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 -261> 2013-04-03 10:50:06.862377 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:0/2151250553 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x25fb400 con 0x3521420 -260> 2013-04-03 10:50:06.862392 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11042 10.252.24.25:0/2614196857 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (432698733 0 0) 0x287ee00 con 0x35212c0 -259> 2013-04-03 10:50:06.862417 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -258> 2013-04-03 10:50:06.862431 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.11042 10.252.24.25:0/2614196857 -257> 2013-04-03 10:50:06.862441 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 -256> 2013-04-03 10:50:06.862549 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:0/2614196857 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x25fb800 con 0x35212c0 -255> 2013-04-03 10:50:06.862568 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11246 10.252.24.25:0/1003678 1 ==== auth(proto 0 43 bytes epoch 5) v1 ==== 73+0+0 (838084839 0 0) 0x1c74800 con 0x3521b00 -254> 2013-04-03 10:50:06.862574 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 do not have session, making new one -253> 2013-04-03 10:50:06.862576 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 ms_dispatch new session MonSession: client.11246 10.252.24.25:0/1003678 is open for client.11246 10.252.24.25:0/1003678 -252> 2013-04-03 10:50:06.862580 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 setting timeout on session -251> 2013-04-03 10:50:06.862601 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -250> 2013-04-03 10:50:06.862615 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 0 43 bytes epoch 5) v1 from client.11246 10.252.24.25:0/1003678 -249> 2013-04-03 10:50:06.862625 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=43 -248> 2013-04-03 10:50:06.862646 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:0/1003678 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x33b8600 con 0x3521b00 -247> 2013-04-03 10:50:06.862657 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11802 10.252.24.30:0/517 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (420838440 0 0) 0x2b76000 con 0x35219a0 -246> 2013-04-03 10:50:06.862679 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -245> 2013-04-03 10:50:06.862693 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from client.11802 10.252.24.30:0/517 -244> 2013-04-03 10:50:06.862702 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 -243> 2013-04-03 10:50:06.862817 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:0/517 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x1c74800 con 0x35219a0 -242> 2013-04-03 10:50:06.862832 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.0 10.252.24.25:6802/3144 3 ==== mon_subscribe({mdsmap=189+,monmap=6+,osdmap=609}) v2 ==== 61+0+0 (2193682605 0 0) 0x28a51c0 con 0x3521840 -241> 2013-04-03 10:50:06.862840 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+,osdmap=609}) v2 -240> 2013-04-03 10:50:06.862847 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -239> 2013-04-03 10:50:06.862851 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mds.0 10.252.24.25:6802/3144 -- mon_subscribe_ack(300s) v1 -- ?+0 0x28a5a80 -238> 2013-04-03 10:50:06.862860 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.2 10.252.24.30:6801/5656 2 ==== auth(proto 2 128 bytes epoch 0) v1 ==== 158+0+0 (1571779039 0 0) 0x1c74600 con 0x35216e0 -237> 2013-04-03 10:50:06.862884 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -236> 2013-04-03 10:50:06.862899 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 128 bytes epoch 0) v1 from osd.2 10.252.24.30:6801/5656 -235> 2013-04-03 10:50:06.862909 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=128 -234> 2013-04-03 10:50:06.863289 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6801/5656 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x2b76000 con 0x35216e0 -233> 2013-04-03 10:50:06.863319 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.? 10.252.24.25:0/5246 3 ==== mon_subscribe({log-info=394162+,monmap=6+}) v2 ==== 44+0+0 (559912796 0 0) 0x2e8be00 con 0x3521580 -232> 2013-04-03 10:50:06.863328 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({log-info=394162+,monmap=6+}) v2 -231> 2013-04-03 10:50:06.863333 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 check_sub client wants log-info ver 394162 -230> 2013-04-03 10:50:06.863354 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 check_sub client client.? 10.252.24.25:0/5246 requested version (394162) is greater than ours (394161), which means we already sent him everything we have. -229> 2013-04-03 10:50:06.863366 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -228> 2013-04-03 10:50:06.863369 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.? 10.252.24.25:0/5246 -- mon_subscribe_ack(300s) v1 -- ?+0 0x28a51c0 -227> 2013-04-03 10:50:06.863379 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11045 10.252.24.25:0/2151250553 3 ==== mon_subscribe({mdsmap=189+,monmap=6+}) v2 ==== 42+0+0 (3152986420 0 0) 0x2e8b540 con 0x3521420 -226> 2013-04-03 10:50:06.863385 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 -225> 2013-04-03 10:50:06.863389 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -224> 2013-04-03 10:50:06.863391 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11045 10.252.24.25:0/2151250553 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8be00 -223> 2013-04-03 10:50:06.863398 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11042 10.252.24.25:0/2614196857 3 ==== mon_subscribe({mdsmap=189+,monmap=6+}) v2 ==== 42+0+0 (3152986420 0 0) 0x2e8b8c0 con 0x35212c0 -222> 2013-04-03 10:50:06.863404 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 -221> 2013-04-03 10:50:06.863408 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -220> 2013-04-03 10:50:06.863409 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11042 10.252.24.25:0/2614196857 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8b540 -219> 2013-04-03 10:50:06.863417 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11246 10.252.24.25:0/1003678 2 ==== auth(proto 2 144 bytes epoch 0) v1 ==== 174+0+0 (4130021166 0 0) 0x1c74800 con 0x3521b00 -218> 2013-04-03 10:50:06.863443 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -217> 2013-04-03 10:50:06.863460 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 preprocess_query auth(proto 2 144 bytes epoch 0) v1 from client.11246 10.252.24.25:0/1003678 -216> 2013-04-03 10:50:06.863470 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 prep_auth() blob_size=144 -215> 2013-04-03 10:50:06.863584 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:0/1003678 -- auth_reply(proto 2 0 Success) v1 -- ?+0 0x1c74600 con 0x3521b00 -214> 2013-04-03 10:50:06.863599 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11802 10.252.24.30:0/517 3 ==== mon_subscribe({mdsmap=189+,monmap=6+}) v2 ==== 42+0+0 (3152986420 0 0) 0x2e8b1c0 con 0x35219a0 -213> 2013-04-03 10:50:06.863606 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({mdsmap=189+,monmap=6+}) v2 -212> 2013-04-03 10:50:06.863611 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -211> 2013-04-03 10:50:06.863613 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11802 10.252.24.30:0/517 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8b8c0 -210> 2013-04-03 10:50:06.863621 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.? 10.252.24.25:0/5246 4 ==== mon_command(status v 0) v1 ==== 48+0+0 (2169389815 0 0) 0x25fba00 con 0x3521580 -209> 2013-04-03 10:50:06.863631 7fcdffcdb700 0 mon.hotcpc9039@0(leader) e5 handle_command mon_command(status v 0) v1 -208> 2013-04-03 10:50:06.863701 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:0/5246 -- mon_command_ack([status]=0 health HEALTH_OK monmap e5: 3 mons at {hotcpc12533=10.252.24.88:6789/0,hotcpc9039=10.252.24.25:6789/0,hotcpc9882=10.252.24.30:6789/0}, election epoch 356, quorum 0,1,2 hotcpc12533,hotcpc9039,hotcpc9882 osdmap e608: 5 osds: 5 up, 5 in pgmap v29179: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail mdsmap e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby v0) v1 -- ?+0 0x1c74800 con 0x3521580 -207> 2013-04-03 10:50:06.863867 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.2 10.252.24.30:6801/5656 3 ==== mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=607}) v2 ==== 69+0+0 (3841848226 0 0) 0x2e8b000 con 0x35216e0 -206> 2013-04-03 10:50:06.863880 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=607}) v2 -205> 2013-04-03 10:50:06.863886 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -204> 2013-04-03 10:50:06.863891 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e608 send_incremental [607..608] to osd.2 10.252.24.30:6801/5656 -203> 2013-04-03 10:50:06.863913 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 build_incremental [607..608] -202> 2013-04-03 10:50:06.863945 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.2 10.252.24.30:6801/5656 -- osd_map(607..608 src has 107..608) v3 -- ?+0 0x25fba00 -201> 2013-04-03 10:50:06.863960 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.2 10.252.24.30:6801/5656 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8b1c0 -200> 2013-04-03 10:50:06.863976 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.2 10.252.24.30:6801/5656 4 ==== mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=607}) v2 ==== 69+0+0 (3841848226 0 0) 0x336b540 con 0x35216e0 -199> 2013-04-03 10:50:06.863984 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=607}) v2 -198> 2013-04-03 10:50:06.863990 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -197> 2013-04-03 10:50:06.863994 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e608 send_incremental [607..608] to osd.2 10.252.24.30:6801/5656 -196> 2013-04-03 10:50:06.864010 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 build_incremental [607..608] -195> 2013-04-03 10:50:06.864038 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.2 10.252.24.30:6801/5656 -- osd_map(607..608 src has 107..608) v3 -- ?+0 0x1c75600 -194> 2013-04-03 10:50:06.864049 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.2 10.252.24.30:6801/5656 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8b000 -193> 2013-04-03 10:50:06.864188 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== client.11246 10.252.24.25:0/1003678 3 ==== mon_subscribe({monmap=6+,osdmap=609}) v2 ==== 42+0+0 (2773195327 0 0) 0x2e8b700 con 0x3521b00 -192> 2013-04-03 10:50:06.864198 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osdmap=609}) v2 -191> 2013-04-03 10:50:06.864203 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -190> 2013-04-03 10:50:06.864206 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> client.11246 10.252.24.25:0/1003678 -- mon_subscribe_ack(300s) v1 -- ?+0 0x336b540 -189> 2013-04-03 10:50:07.605625 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 253 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140535 v188) v2 ==== 289+0+0 (803607707 0 0) 0x20e3b00 con 0x2118420 -188> 2013-04-03 10:50:07.605698 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(10187/hotcpc12974 up:standby seq 140535 v188) v2 from mds.? 10.252.58.31:6800/10908 -187> 2013-04-03 10:50:07.605711 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.58.31:6800/10908 -- mdsbeacon(10187/hotcpc12974 up:standby seq 140535 v188) v2 -- ?+0 0x2cc58c0 con 0x2118420 -186> 2013-04-03 10:50:07.696642 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.0 10.252.58.31:6802/2682 13 ==== mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=606}) v2 ==== 69+0+0 (281224682 0 0) 0x336b000 con 0x3311580 -185> 2013-04-03 10:50:07.696656 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_subscribe mon_subscribe({monmap=6+,osd_pg_creates=0,osdmap=606}) v2 -184> 2013-04-03 10:50:07.696663 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 check_sub monmap next 6 have 5 -183> 2013-04-03 10:50:07.696669 7fcdffcdb700 5 mon.hotcpc9039@0(leader).osd e608 send_incremental [606..608] to osd.0 10.252.58.31:6802/2682 -182> 2013-04-03 10:50:07.696690 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 build_incremental [606..608] -181> 2013-04-03 10:50:07.696728 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.0 10.252.58.31:6802/2682 -- osd_map(606..608 src has 107..608) v3 -- ?+0 0x33e9000 -180> 2013-04-03 10:50:07.696741 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> osd.0 10.252.58.31:6802/2682 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2e8b700 -179> 2013-04-03 10:50:08.397341 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29179 check_down_pgs -178> 2013-04-03 10:50:08.397368 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29179 v29179: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail -177> 2013-04-03 10:50:08.397396 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -176> 2013-04-03 10:50:08.397418 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -175> 2013-04-03 10:50:08.397455 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 min_last_epoch_clean 594 -174> 2013-04-03 10:50:08.397486 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos -173> 2013-04-03 10:50:08.397505 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 -172> 2013-04-03 10:50:08.397514 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 log -171> 2013-04-03 10:50:08.397535 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -170> 2013-04-03 10:50:08.397554 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -169> 2013-04-03 10:50:08.815564 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427637 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3645280 -168> 2013-04-03 10:50:08.815603 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(lease lc 427637 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3645000 -167> 2013-04-03 10:50:08.815870 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).connect error 10.252.24.88:6789/0, 111: Connection refused -166> 2013-04-03 10:50:08.815899 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).fault 111: Connection refused -165> 2013-04-03 10:50:08.815907 7fce027bf700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).fault -164> 2013-04-03 10:50:08.816083 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).connect error 10.252.24.88:6789/0, 111: Connection refused -163> 2013-04-03 10:50:08.816111 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).fault 111: Connection refused -162> 2013-04-03 10:50:08.816320 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 117 ==== paxos(lease_ack lc 427637 fc 427608 pn 0 opn 0) v3 ==== 80+0+0 (1768918616 0 0) 0x3645280 con 0x208a9a0 -161> 2013-04-03 10:50:08.816389 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos -160> 2013-04-03 10:50:08.816409 7fcdffcdb700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 -159> 2013-04-03 10:50:08.816428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -158> 2013-04-03 10:50:08.862287 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 18 ==== pg_stats(1 pgs tid 7 v 608) v1 ==== 485+0+0 (3362394779 0 0) 0x20fc900 con 0x33112c0 -157> 2013-04-03 10:50:08.862340 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 preprocess_query pg_stats(1 pgs tid 7 v 608) v1 from osd.3 10.252.24.88:6804/18376 -156> 2013-04-03 10:50:08.862362 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 prepare_update pg_stats(1 pgs tid 7 v 608) v1 from osd.3 10.252.24.88:6804/18376 -155> 2013-04-03 10:50:08.862367 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 prepare_pg_stats pg_stats(1 pgs tid 7 v 608) v1 from osd.3 -154> 2013-04-03 10:50:08.862373 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29179 got osd.3 osd_stat(1182 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[]) (was osd_stat(1185 MB used, 356 GB avail, 359 GB total, peers [0,1,2,4]/[])) -153> 2013-04-03 10:50:08.912574 7fce004dc700 10 mon.hotcpc9039@0(leader).pg v29179 encode_pending v 29180 -152> 2013-04-03 10:50:08.912618 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos active c 427608..427637) queue_proposal bl 654 bytes; ctx = 0x2c3d1f0 -151> 2013-04-03 10:50:08.912634 7fce004dc700 5 mon.hotcpc9039@0(leader).paxos(paxos preparing update c 427608..427637) propose_queued 427638 654 bytes -150> 2013-04-03 10:50:08.913674 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(begin lc 427637 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3476280 -149> 2013-04-03 10:50:08.913710 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(begin lc 427637 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3450500 -148> 2013-04-03 10:50:08.913914 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).connect error 10.252.24.88:6789/0, 111: Connection refused -147> 2013-04-03 10:50:08.913948 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).fault 111: Connection refused -146> 2013-04-03 10:50:08.951377 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.0 10.252.24.25:6802/3144 4 ==== mdsbeacon(11234/hotcpc9039 up:active seq 250 v188) v2 ==== 288+0+0 (2917908214 0 0) 0x20e4b80 con 0x3521840 -145> 2013-04-03 10:50:08.951428 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 preprocess_query mdsbeacon(11234/hotcpc9039 up:active seq 250 v188) v2 from mds.0 10.252.24.25:6802/3144 -144> 2013-04-03 10:50:08.951440 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.25:6802/3144 -- mdsbeacon(11234/hotcpc9039 up:active seq 250 v188) v2 -- ?+0 0x20e3b00 con 0x3521840 -143> 2013-04-03 10:50:08.955336 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 118 ==== paxos(accept lc 427637 fc 0 pn 17800 opn 0) v3 ==== 80+0+0 (2133131114 0 0) 0x3476280 con 0x208a9a0 -142> 2013-04-03 10:50:08.963277 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(commit lc 427638 fc 0 pn 17800 opn 0) v3 -- ?+0 0x3645280 -141> 2013-04-03 10:50:08.963303 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- paxos(commit lc 427638 fc 0 pn 17800 opn 0) v3 -- ?+0 0x348c280 -140> 2013-04-03 10:50:08.963467 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).connect error 10.252.24.88:6789/0, 111: Connection refused -139> 2013-04-03 10:50:08.963496 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).fault 111: Connection refused -138> 2013-04-03 10:50:09.440877 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 49 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 41 v188) v2 ==== 289+0+0 (136808360 0 0) 0x3614840 con 0x3311000 -137> 2013-04-03 10:50:09.440956 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427638) is_readable now=2013-04-03 10:50:09.440957 lease_expire=0.000000 has v0 lc 427638 -136> 2013-04-03 10:50:09.661878 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 24 ==== pg_stats(1 pgs tid 85 v 608) v1 ==== 477+0+0 (3730641380 0 0) 0x363f900 con 0x3311160 -135> 2013-04-03 10:50:09.763738 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).connect error 10.252.24.88:6789/0, 111: Connection refused -134> 2013-04-03 10:50:09.763770 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).fault 111: Connection refused -133> 2013-04-03 10:50:11.364052 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).connect error 10.252.24.88:6789/0, 111: Connection refused -132> 2013-04-03 10:50:11.364091 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).fault 111: Connection refused -131> 2013-04-03 10:50:11.605715 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 254 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140536 v188) v2 ==== 289+0+0 (3648431601 0 0) 0x20e4340 con 0x2118420 -130> 2013-04-03 10:50:11.605794 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427638) is_readable now=2013-04-03 10:50:11.605795 lease_expire=0.000000 has v0 lc 427638 -129> 2013-04-03 10:50:11.609192 7fce026be700 1 -- 10.252.24.25:6789/0 >> :/0 pipe(0x3657780 sd=38 :6789 s=0 pgs=0 cs=0 l=0).accept sd=38 10.252.24.88:33225/0 -128> 2013-04-03 10:50:11.609209 7fce026be700 0 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3657780 sd=38 :6789 s=0 pgs=0 cs=0 l=0).accept peer addr is really 10.252.24.88:0/3253115431 (socket is 10.252.24.88:33225/0) -127> 2013-04-03 10:50:11.609226 7fce026be700 1 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3657780 sd=38 :6789 s=0 pgs=0 cs=0 l=1).peer missing required features 2040000 -126> 2013-04-03 10:50:11.609443 7fce026be700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:0/3253115431 pipe(0x3657780 sd=38 :6789 s=4 pgs=0 cs=0 l=1).fault 0: Success -125> 2013-04-03 10:50:12.951515 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.0 10.252.24.25:6802/3144 5 ==== mdsbeacon(11234/hotcpc9039 up:active seq 251 v188) v2 ==== 288+0+0 (1894680261 0 0) 0x1bae8c0 con 0x3521840 -124> 2013-04-03 10:50:12.951597 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427638) is_readable now=2013-04-03 10:50:12.951598 lease_expire=0.000000 has v0 lc 427638 -123> 2013-04-03 10:50:13.397706 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -122> 2013-04-03 10:50:13.397742 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -121> 2013-04-03 10:50:13.397764 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos -120> 2013-04-03 10:50:13.397783 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 -119> 2013-04-03 10:50:13.397791 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 log -118> 2013-04-03 10:50:13.397809 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -117> 2013-04-03 10:50:13.397828 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -116> 2013-04-03 10:50:13.440906 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 50 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 42 v188) v2 ==== 289+0+0 (1838433839 0 0) 0x3614580 con 0x3311000 -115> 2013-04-03 10:50:13.440963 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427638) is_readable now=2013-04-03 10:50:13.440964 lease_expire=0.000000 has v0 lc 427638 -114> 2013-04-03 10:50:13.862955 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 19 ==== pg_stats(1 pgs tid 8 v 608) v1 ==== 485+0+0 (2584050952 0 0) 0x20adb40 con 0x33112c0 -113> 2013-04-03 10:50:14.564357 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).connect error 10.252.24.88:6789/0, 111: Connection refused -112> 2013-04-03 10:50:14.564392 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).fault 111: Connection refused -111> 2013-04-03 10:50:14.662574 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 25 ==== pg_stats(1 pgs tid 86 v 608) v1 ==== 477+0+0 (2033325136 0 0) 0x363f6c0 con 0x3311160 -110> 2013-04-03 10:50:15.307304 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.2 10.252.24.30:6801/5656 5 ==== pg_stats(1 pgs tid 66 v 608) v1 ==== 485+0+0 (4216735155 0 0) 0x20fc240 con 0x35216e0 -109> 2013-04-03 10:50:15.605848 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 255 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140537 v188) v2 ==== 289+0+0 (4195883660 0 0) 0x20e3dc0 con 0x2118420 -108> 2013-04-03 10:50:15.605916 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427638) is_readable now=2013-04-03 10:50:15.605918 lease_expire=0.000000 has v0 lc 427638 -107> 2013-04-03 10:50:16.951657 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.0 10.252.24.25:6802/3144 6 ==== mdsbeacon(11234/hotcpc9039 up:active seq 252 v188) v2 ==== 288+0+0 (1747829694 0 0) 0x1bae080 con 0x3521840 -106> 2013-04-03 10:50:16.951737 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427638) is_readable now=2013-04-03 10:50:16.951738 lease_expire=0.000000 has v0 lc 427638 -105> 2013-04-03 10:50:17.441112 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 51 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 43 v188) v2 ==== 289+0+0 (1325070674 0 0) 0x36142c0 con 0x3311000 -104> 2013-04-03 10:50:17.441197 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos updating c 427608..427638) is_readable now=2013-04-03 10:50:17.441198 lease_expire=0.000000 has v0 lc 427638 -103> 2013-04-03 10:50:18.397970 7fce004dc700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -102> 2013-04-03 10:50:18.398012 7fce004dc700 10 mon.hotcpc9039@0(leader).osd e608 e608: 5 osds: 5 up, 5 in -101> 2013-04-03 10:50:18.398034 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos -100> 2013-04-03 10:50:18.398066 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 update_from_paxos version 394161 summary v 394161 -99> 2013-04-03 10:50:18.398075 7fce004dc700 10 mon.hotcpc9039@0(leader).log v394161 log -98> 2013-04-03 10:50:18.398093 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 update_from_paxos -97> 2013-04-03 10:50:18.398112 7fce004dc700 10 mon.hotcpc9039@0(leader).auth v2063 auth -96> 2013-04-03 10:50:18.816446 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 119 ==== mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 ==== 95+0+0 (3455676706 0 0) 0x3516900 con 0x208a9a0 -95> 2013-04-03 10:50:18.816470 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_probe mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -94> 2013-04-03 10:50:18.816482 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 handle_probe_probe mon.1 10.252.24.30:6789/0mon_probe(probe d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9882) v4 -93> 2013-04-03 10:50:18.816492 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> 10.252.24.30:6789/0 -- mon_probe(reply d6e91ff1-9b43-4972-b7a7-a22d84497d78 name hotcpc9039 quorum 0,1,2 paxos( fc 427608 lc 427638 )) v4 -- ?+0 0x3516000 con 0x208a9a0 -92> 2013-04-03 10:50:18.863385 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 20 ==== pg_stats(1 pgs tid 9 v 608) v1 ==== 485+0+0 (3125154819 0 0) 0x20ad900 con 0x33112c0 -91> 2013-04-03 10:50:18.907144 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 120 ==== election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 propose 357) v3 ==== 537+0+0 (2772686900 0 0) 0x20fc480 con 0x208a9a0 -90> 2013-04-03 10:50:18.907166 7fcdffcdb700 5 mon.hotcpc9039@0(leader).elector(356) handle_propose from mon.1 -89> 2013-04-03 10:50:18.907170 7fcdffcdb700 10 mon.hotcpc9039@0(leader).elector(356) bump_epoch 356 to 357 -88> 2013-04-03 10:50:18.908377 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 reset -87> 2013-04-03 10:50:18.908392 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish -86> 2013-04-03 10:50:18.908465 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 start_election -85> 2013-04-03 10:50:18.908470 7fcdffcdb700 10 mon.hotcpc9039@0(leader) e5 cancel_probe_timeout (none scheduled) -84> 2013-04-03 10:50:18.908476 7fcdffcdb700 0 log [INF] : mon.hotcpc9039 calling new monitor election -83> 2013-04-03 10:50:18.908477 7fcdffcdb700 10 send_log to self -82> 2013-04-03 10:50:18.908478 7fcdffcdb700 10 log_queue is 2 last_log 263 sent 261 num 2 unsent 2 sending 2 -81> 2013-04-03 10:50:18.908480 7fcdffcdb700 10 will send 2013-04-03 10:50:05.778345 mon.0 10.252.24.25:6789/0 262 : [INF] pgmap v29179: 240 pgs: 240 active+clean; 256 MB data, 5668 MB used, 1067 GB / 1079 GB avail -80> 2013-04-03 10:50:18.908487 7fcdffcdb700 10 will send 2013-04-03 10:50:18.908477 mon.0 10.252.24.25:6789/0 263 : [INF] mon.hotcpc9039 calling new monitor election -79> 2013-04-03 10:50:18.908491 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x33b8800 -78> 2013-04-03 10:50:18.908497 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(357) start -- can i be leader? -77> 2013-04-03 10:50:18.908512 7fcdffcdb700 1 mon.hotcpc9039@0(electing).elector(357) init, last seen epoch 357 -76> 2013-04-03 10:50:18.908520 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 propose 357) v3 -- ?+0 0x363fd80 -75> 2013-04-03 10:50:18.908533 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.2 10.252.24.88:6789/0 -- election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 propose 357) v3 -- ?+0 0x35e0240 -74> 2013-04-03 10:50:18.908546 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(2 entries) v1 ==== 0+0+0 (0 0 0) 0x33b8800 con 0x1bac160 -73> 2013-04-03 10:50:18.908577 7fcdffcdb700 1 mon.hotcpc9039@0(electing).paxos(paxos recovering c 427608..427638) is_readable now=2013-04-03 10:50:18.908578 lease_expire=0.000000 has v0 lc 427638 -72> 2013-04-03 10:50:18.908702 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).connect error 10.252.24.88:6789/0, 111: Connection refused -71> 2013-04-03 10:50:18.908737 7fce027bf700 2 -- 10.252.24.25:6789/0 >> 10.252.24.88:6789/0 pipe(0x3303a00 sd=28 :6789 s=1 pgs=151512 cs=2 l=0).fault 111: Connection refused -70> 2013-04-03 10:50:18.909212 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 121 ==== election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 ack 357) v3 ==== 537+0+0 (1131630170 0 0) 0x34d9d80 con 0x208a9a0 -69> 2013-04-03 10:50:18.909228 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(357) handle_ack from mon.1 -68> 2013-04-03 10:50:18.909231 7fcdffcdb700 5 mon.hotcpc9039@0(electing).elector(357) so far i have {0=4294967295,1=4294967295} -67> 2013-04-03 10:50:19.605997 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 256 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140538 v188) v2 ==== 289+0+0 (2678840075 0 0) 0x20e3840 con 0x2118420 -66> 2013-04-03 10:50:19.606076 7fcdffcdb700 1 mon.hotcpc9039@0(electing).paxos(paxos recovering c 427608..427638) is_readable now=2013-04-03 10:50:19.606077 lease_expire=0.000000 has v0 lc 427638 -65> 2013-04-03 10:50:19.663103 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.1 10.252.24.25:6805/3572 26 ==== pg_stats(1 pgs tid 87 v 608) v1 ==== 477+0+0 (1701109194 0 0) 0x363f480 con 0x3311160 -64> 2013-04-03 10:50:20.307777 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.2 10.252.24.30:6801/5656 6 ==== pg_stats(1 pgs tid 67 v 608) v1 ==== 485+0+0 (1179694211 0 0) 0x35e0480 con 0x35216e0 -63> 2013-04-03 10:50:20.951780 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.0 10.252.24.25:6802/3144 7 ==== mdsbeacon(11234/hotcpc9039 up:active seq 253 v188) v2 ==== 288+0+0 (3039361933 0 0) 0x2ce1580 con 0x3521840 -62> 2013-04-03 10:50:20.951852 7fcdffcdb700 1 mon.hotcpc9039@0(electing).paxos(paxos recovering c 427608..427638) is_readable now=2013-04-03 10:50:20.951853 lease_expire=0.000000 has v0 lc 427638 -61> 2013-04-03 10:50:21.441191 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.24.88:6800/18040 52 ==== mdsbeacon(11904/hotcpc12533 up:standby seq 44 v188) v2 ==== 289+0+0 (2800993569 0 0) 0x3614000 con 0x3311000 -60> 2013-04-03 10:50:21.441270 7fcdffcdb700 1 mon.hotcpc9039@0(electing).paxos(paxos recovering c 427608..427638) is_readable now=2013-04-03 10:50:21.441271 lease_expire=0.000000 has v0 lc 427638 -59> 2013-04-03 10:50:23.606112 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mds.? 10.252.58.31:6800/10908 257 ==== mdsbeacon(10187/hotcpc12974 up:standby seq 140539 v188) v2 ==== 289+0+0 (3167052918 0 0) 0x20e4080 con 0x2118420 -58> 2013-04-03 10:50:23.606172 7fcdffcdb700 1 mon.hotcpc9039@0(electing).paxos(paxos recovering c 427608..427638) is_readable now=2013-04-03 10:50:23.606174 lease_expire=0.000000 has v0 lc 427638 -57> 2013-04-03 10:50:23.863959 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== osd.3 10.252.24.88:6804/18376 21 ==== pg_stats(1 pgs tid 10 v 608) v1 ==== 485+0+0 (2436778021 0 0) 0x20ad6c0 con 0x33112c0 -56> 2013-04-03 10:50:23.908610 7fce004dc700 5 mon.hotcpc9039@0(electing).elector(357) election timer expired -55> 2013-04-03 10:50:23.908627 7fce004dc700 10 mon.hotcpc9039@0(electing).elector(357) bump_epoch 357 to 358 -54> 2013-04-03 10:50:23.908938 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- election(d6e91ff1-9b43-4972-b7a7-a22d84497d78 victory 358) v3 -- ?+0 0x24f8900 -53> 2013-04-03 10:50:23.908956 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 win_election, epoch 358 quorum is 0,1 features are 4294967295 -52> 2013-04-03 10:50:23.908966 7fce004dc700 0 log [INF] : mon.hotcpc9039@0 won leader election with quorum 0,1 -51> 2013-04-03 10:50:23.908967 7fce004dc700 10 send_log to self -50> 2013-04-03 10:50:23.908968 7fce004dc700 10 log_queue is 3 last_log 264 sent 263 num 3 unsent 1 sending 1 -49> 2013-04-03 10:50:23.908970 7fce004dc700 10 will send 2013-04-03 10:50:18.908477 mon.0 10.252.24.25:6789/0 263 : [INF] mon.hotcpc9039 calling new monitor election -48> 2013-04-03 10:50:23.908976 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(1 entries) v1 -- ?+0 0x1c73e00 -47> 2013-04-03 10:50:23.908990 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.0 10.252.24.25:6789/0 0 ==== log(1 entries) v1 ==== 0+0+0 (0 0 0) 0x1c73e00 con 0x1bac160 -46> 2013-04-03 10:50:23.909199 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(collect lc 427638 fc 427608 pn 17900 opn 0) v3 -- ?+0 0x3450000 -45> 2013-04-03 10:50:23.909279 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427608..427638) is_readable now=2013-04-03 10:50:23.909280 lease_expire=0.000000 has v0 lc 427638 -44> 2013-04-03 10:50:23.909304 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427608..427638) is_readable now=2013-04-03 10:50:23.909305 lease_expire=0.000000 has v0 lc 427638 -43> 2013-04-03 10:50:23.909341 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427608..427638) is_readable now=2013-04-03 10:50:23.909342 lease_expire=0.000000 has v0 lc 427638 -42> 2013-04-03 10:50:23.909364 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427608..427638) is_readable now=2013-04-03 10:50:23.909365 lease_expire=0.000000 has v0 lc 427638 -41> 2013-04-03 10:50:23.909386 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427608..427638) is_readable now=2013-04-03 10:50:23.909389 lease_expire=0.000000 has v0 lc 427638 -40> 2013-04-03 10:50:23.909414 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427608..427638) is_readable now=2013-04-03 10:50:23.909415 lease_expire=0.000000 has v0 lc 427638 -39> 2013-04-03 10:50:23.909438 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427608..427638) is_readable now=2013-04-03 10:50:23.909439 lease_expire=0.000000 has v0 lc 427638 -38> 2013-04-03 10:50:23.909461 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427608..427638) is_readable now=2013-04-03 10:50:23.909462 lease_expire=0.000000 has v0 lc 427638 -37> 2013-04-03 10:50:23.909485 7fce004dc700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427608..427638) is_readable now=2013-04-03 10:50:23.909486 lease_expire=0.000000 has v0 lc 427638 -36> 2013-04-03 10:50:23.909494 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_finish -35> 2013-04-03 10:50:23.909497 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 resend_routed_requests -34> 2013-04-03 10:50:23.909499 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 register_cluster_logger - already registered -33> 2013-04-03 10:50:23.909501 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start -32> 2013-04-03 10:50:23.909502 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round curr 0 -31> 2013-04-03 10:50:23.909504 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round new 1 -30> 2013-04-03 10:50:23.909505 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck -29> 2013-04-03 10:50:23.909506 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck start timecheck epoch 358 round 1 -28> 2013-04-03 10:50:23.909510 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck send time_check( ping e 358 r 1 ) v1 to mon.1 10.252.24.30:6789/0 -27> 2013-04-03 10:50:23.909515 7fce004dc700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- time_check( ping e 358 r 1 ) v1 -- ?+0 0x24f8480 -26> 2013-04-03 10:50:23.909523 7fce004dc700 10 mon.hotcpc9039@0(leader) e5 timecheck_start_round setting up next event -25> 2013-04-03 10:50:23.909623 7fcdffcdb700 1 mon.hotcpc9039@0(leader).paxos(paxos recovering c 427608..427638) is_readable now=2013-04-03 10:50:23.909627 lease_expire=0.000000 has v0 lc 427638 -24> 2013-04-03 10:50:23.979416 7fcdffcdb700 1 -- 10.252.24.25:6789/0 <== mon.1 10.252.24.30:6789/0 122 ==== paxos(last lc 427638 fc 427608 pn 17900 opn 0) v3 ==== 80+0+0 (2015154 0 0) 0x3450000 con 0x208a9a0 -23> 2013-04-03 10:50:23.979449 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.1 10.252.24.30:6789/0 -- paxos(lease lc 427638 fc 427608 pn 0 opn 0) v3 -- ?+0 0x3476280 -22> 2013-04-03 10:50:23.979493 7fcdffcdb700 7 mon.hotcpc9039@0(leader).pg v29179 update_from_paxos applying incremental 29180 -21> 2013-04-03 10:50:23.979512 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29180 v29180: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 243B/s wr, 0op/s -20> 2013-04-03 10:50:23.979530 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29180 send_pg_creates to 0 pgs -19> 2013-04-03 10:50:23.979533 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29180 update_logger -18> 2013-04-03 10:50:23.979543 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29180 create_pending v 29181 -17> 2013-04-03 10:50:23.979551 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29180 check_osd_map already seen 608 >= 608 -16> 2013-04-03 10:50:23.979553 7fcdffcdb700 10 mon.hotcpc9039@0(leader).pg v29180 update_logger -15> 2013-04-03 10:50:23.979562 7fcdffcdb700 0 log [INF] : pgmap v29180: 240 pgs: 240 active+clean; 256 MB data, 5665 MB used, 1067 GB / 1079 GB avail; 243B/s wr, 0op/s -14> 2013-04-03 10:50:23.979563 7fcdffcdb700 10 send_log to self -13> 2013-04-03 10:50:23.979564 7fcdffcdb700 10 log_queue is 4 last_log 265 sent 263 num 4 unsent 2 sending 2 -12> 2013-04-03 10:50:23.979565 7fcdffcdb700 10 will send 2013-04-03 10:50:18.908477 mon.0 10.252.24.25:6789/0 263 : [INF] mon.hotcpc9039 calling new monitor election -11> 2013-04-03 10:50:23.979571 7fcdffcdb700 10 will send 2013-04-03 10:50:23.908967 mon.0 10.252.24.25:6789/0 264 : [INF] mon.hotcpc9039@0 won leader election with quorum 0,1 -10> 2013-04-03 10:50:23.979574 7fcdffcdb700 1 -- 10.252.24.25:6789/0 --> mon.0 10.252.24.25:6789/0 -- log(2 entries) v1 -- ?+0 0x33b8a00 -9> 2013-04-03 10:50:23.979605 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 create_pending e189 -8> 2013-04-03 10:50:23.979625 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 e188: 1/1/1 up {0=hotcpc9039=up:active}, 2 up:standby -7> 2013-04-03 10:50:23.979634 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 no beacon from 10187 10.252.58.31:6800/10908 mds.-1.0 up:standby since 2013-04-03 10:50:07.605709 -6> 2013-04-03 10:50:23.979644 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 removing 10187 10.252.58.31:6800/10908 mds.-1.0 up:standby (laggy) -5> 2013-04-03 10:50:23.979650 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 no beacon from 11234 10.252.24.25:6802/3144 mds.0.29 up:active since 2013-04-03 10:50:08.951439 -4> 2013-04-03 10:50:23.979654 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 replacing 11234 10.252.24.25:6802/3144 mds.0.29 up:active with 11904/hotcpc12533 10.252.24.88:6800/18040 -3> 2013-04-03 10:50:23.979659 7fcdffcdb700 10 mon.hotcpc9039@0(leader).osd e608 blacklist 10.252.24.25:6802/3144 until 2013-04-03 11:14:23.979632 -2> 2013-04-03 10:50:23.979664 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 no beacon from 11904 10.252.24.88:6800/18040 mds.0.30 up:replay since 2013-04-03 10:50:05.440875 -1> 2013-04-03 10:50:23.979668 7fcdffcdb700 10 mon.hotcpc9039@0(leader).mds e188 marking 11904 10.252.24.88:6800/18040 mds.0.30 up:replay laggy 0> 2013-04-03 10:50:23.980641 7fcdffcdb700 -1 mon/PaxosService.cc: In function 'void PaxosService::propose_pending()' thread 7fcdffcdb700 time 2013-04-03 10:50:23.979673 mon/PaxosService.cc: 127: FAILED assert(have_pending) ceph version 0.59 (cbae6a435c62899f857775f66659de052fb0e759) 1: (PaxosService::propose_pending()+0xbe3) [0x4d7043] 2: (MDSMonitor::tick()+0x2dce) [0x5127ee] 3: (MDSMonitor::on_active()+0x1f) [0x50d89f] 4: (PaxosService::_active()+0x2bf) [0x4d743f] 5: (Context::complete(int)+0xa) [0x4ac1da] 6: (finish_contexts(CephContext*, std::list >&, int)+0x106) [0x4af196] 7: (Paxos::handle_last(MMonPaxos*)+0x9d0) [0x4d03c0] 8: (Paxos::dispatch(PaxosServiceMessage*)+0x27b) [0x4d135b] 9: (Monitor::_ms_dispatch(Message*)+0x136a) [0x4ab30a] 10: (Monitor::ms_dispatch(Message*)+0x32) [0x4c40e2] 11: (DispatchQueue::entry()+0x311) [0x686d11] 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x682aad] 13: (()+0x68ca) [0x7fce057df8ca] 14: (clone()+0x6d) [0x7fce044d686d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 10/10 mon 0/10 monc 0/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 hadoop 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 100000 max_new 1000 log_file /var/log/ceph/ceph-mon.hotcpc9039.log --- end dump of recent events --- 2013-04-03 10:50:24.220088 7fcdffcdb700 -1 *** Caught signal (Aborted) ** in thread 7fcdffcdb700 ceph version 0.59 (cbae6a435c62899f857775f66659de052fb0e759) 1: /usr/bin/ceph-mon() [0x57daf9] 2: (()+0xeff0) [0x7fce057e7ff0] 3: (gsignal()+0x35) [0x7fce044391b5] 4: (abort()+0x180) [0x7fce0443bfc0] 5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7fce04ccddc5] 6: (()+0xcb166) [0x7fce04ccc166] 7: (()+0xcb193) [0x7fce04ccc193] 8: (()+0xcb28e) [0x7fce04ccc28e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7c9) [0x5a3579] 10: (PaxosService::propose_pending()+0xbe3) [0x4d7043] 11: (MDSMonitor::tick()+0x2dce) [0x5127ee] 12: (MDSMonitor::on_active()+0x1f) [0x50d89f] 13: (PaxosService::_active()+0x2bf) [0x4d743f] 14: (Context::complete(int)+0xa) [0x4ac1da] 15: (finish_contexts(CephContext*, std::list >&, int)+0x106) [0x4af196] 16: (Paxos::handle_last(MMonPaxos*)+0x9d0) [0x4d03c0] 17: (Paxos::dispatch(PaxosServiceMessage*)+0x27b) [0x4d135b] 18: (Monitor::_ms_dispatch(Message*)+0x136a) [0x4ab30a] 19: (Monitor::ms_dispatch(Message*)+0x32) [0x4c40e2] 20: (DispatchQueue::entry()+0x311) [0x686d11] 21: (DispatchQueue::DispatchThread::entry()+0xd) [0x682aad] 22: (()+0x68ca) [0x7fce057df8ca] 23: (clone()+0x6d) [0x7fce044d686d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2013-04-03 10:50:24.220088 7fcdffcdb700 -1 *** Caught signal (Aborted) ** in thread 7fcdffcdb700 ceph version 0.59 (cbae6a435c62899f857775f66659de052fb0e759) 1: /usr/bin/ceph-mon() [0x57daf9] 2: (()+0xeff0) [0x7fce057e7ff0] 3: (gsignal()+0x35) [0x7fce044391b5] 4: (abort()+0x180) [0x7fce0443bfc0] 5: (__gnu_cxx::__verbose_terminate_handler()+0x115) [0x7fce04ccddc5] 6: (()+0xcb166) [0x7fce04ccc166] 7: (()+0xcb193) [0x7fce04ccc193] 8: (()+0xcb28e) [0x7fce04ccc28e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7c9) [0x5a3579] 10: (PaxosService::propose_pending()+0xbe3) [0x4d7043] 11: (MDSMonitor::tick()+0x2dce) [0x5127ee] 12: (MDSMonitor::on_active()+0x1f) [0x50d89f] 13: (PaxosService::_active()+0x2bf) [0x4d743f] 14: (Context::complete(int)+0xa) [0x4ac1da] 15: (finish_contexts(CephContext*, std::list >&, int)+0x106) [0x4af196] 16: (Paxos::handle_last(MMonPaxos*)+0x9d0) [0x4d03c0] 17: (Paxos::dispatch(PaxosServiceMessage*)+0x27b) [0x4d135b] 18: (Monitor::_ms_dispatch(Message*)+0x136a) [0x4ab30a] 19: (Monitor::ms_dispatch(Message*)+0x32) [0x4c40e2] 20: (DispatchQueue::entry()+0x311) [0x686d11] 21: (DispatchQueue::DispatchThread::entry()+0xd) [0x682aad] 22: (()+0x68ca) [0x7fce057df8ca] 23: (clone()+0x6d) [0x7fce044d686d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 10/10 mon 0/10 monc 0/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 hadoop 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 100000 max_new 1000 log_file /var/log/ceph/ceph-mon.hotcpc9039.log --- end dump of recent events ---